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FOREWORD 


These proceedings contain the papers of the International Conference e-Learning 2015, 
which was organised by the International Association for Development of the Information 
and Society and co-organized by Universidad de Las Palmas de Gran Canaria, Spain, 
21-24 July, 2015. This conference is part of the Multi Conference on Computer Science 
and Information Systems, 21-24 July 2015, which had a total of 652 submissions. 

The e-Learning 2015 conference aims to address the main issues of concern within 
e-Learning. This conference covers both technical as well as the non-technical aspects of 
e-Learning. 

The conference accepted submissions in the following seven main areas: Organisational 
Strategy and Management Issues; Technological Issues; e-Learning Curriculum 
Development Issues; Instructional Design Issues; e-Learning Delivery Issues; e-Learning 
Research Methods and Approaches; e-Skills and Information Literacy for Learning. 

The above referred main submission areas are detailed: 

Organisational Strategy and Management Issues 

- Higher and Further Education 

- Primary and Secondary Education 

- Workplace Learning 

- Vocational Training 

- Home Schooling 

- Distance Learning 

- Blended Learning 

- Change Management 

- Educational Management 

- Continuous Professional Development (CPD) for Educational and Training Staff 

- Return on e-Learning Investments (ROI) 

Technological Issues 

- Learning Management Systems (LMS) 

- Managed Learning Environments (MLEs) 

- Virtual Learning Environments (VLEs) 

- Computer-Mediated Communication (CMC) Tools 

- Social Support Software 

- Architecture of Educational Information Systems Infrastructure 

- Security and Data Protection 

- Learning Objects 

- XML Schemas and the Semantic Web 

- Web 2.0 Applications 


IX 



e-Learning Curriculum Development Issues 

- Philosophies and Epistemologies for e-learning 

- Learning Theories and Approaches for e-learning 

- e-Learning Models 

- Conceptual Representations 

- Pedagogical Models 

- e-Learning Pedagogical Strategies 

- e-Learning Tactics 

- Developing e-Learning for Specific Subject Domains 

Instructional Design Issues 

- Designing e-Learning Settings 

- Developing e-Learning Pilots and Prototypes 

- Creating e-Learning Courses 

- Collaborative learning 

- Problem-based learning 

- Inquiry-based learning 

- Blended learning 

- Distance learning 

- Designing e-Leaming Tasks 

- E-learning activities 

- Online Groupwork 

- Experiential learning 

- Simulations and Modelling 

- Gaming and edutainment 

- Creativity and design activities 

- Exploratory programming 

e-Learning Delivery Issues 

- e-Delivery in different contexts 

- Higher and Further Education 

- Primary and Secondary Schools 

- Workplace Learning 

- Vocational Training 

- Distance Learning 

- Online Assessment 

- Innovations in e-Assessment 

- e-Moderating 

- e-Tutoring 

- e-Facilitating 

- Leadership in e-Learning Delivery 

- Networked Information and Communication Literacy Skills 

- Participation and Motivation in e-Learning 

e-Learning Research Methods and Approaches 

- Action Research 

- Design Research 

- Course and Programme Evaluations 

- Systematic Literature Reviews 

- Historical Analysis 


x 



- Case Studies 

- Meta-analysis of Case Studies 

- Effectiveness and Impact Studies 

- Evaluation of e-Learning Technologies 

- Evaluation of Student and Tutor Satisfaction 

- Learning and cognitive styles 

- Ethical Issues in e-learning 

e-Skills and Information Literacy for Learning 

- Teaching information literacy 

- Electronic library and information search skills 

- ICT skills education 

- in schools and colleges 

- for business, industry and the public sector 

- in adult, community, home and prison education 

- informal methods (peer groups, family) 

- Education for computer-mediated communication skills 

- Netiquette 

- Online safety for children and vulnerable users 

- Cybercrime awareness and personal prevention 

- Student production of online media 

- Web design 

- Digital storytelling 

- Web 2.0 tools 

- etc. 

- Digital media studies 


The e-Learning 2015 conference received 102 submissions from more than 24 countries. 
Each submission has been anonymously reviewed by an average of four independent 
reviewers, to ensure that accepted submissions were of a high standard. Consequently only 
16 full papers were approved which meant an acceptance rate of 16 %. A few more papers 
were accepted as short papers, reflection papers and posters. An extended version of the 
best papers will be selected for publishing as extended versions in the Interactive 
Technology and Smart Education (ITSE) journal (ISSN: 1741-5659) and also in the IADIS 
International Journal on WWW/Internet (ISSN: 1645-7641). Other outlets may also 
receive extended versions of the best papers, including journals from Inderscience. 

Besides the presentation of full papers, short papers, reflection papers and posters, the 
conference also included two keynote presentations from internationally distinguished 
researchers. We would therefore like to express our gratitude to Steven Duggan, Director, 
Worldwide Education Strategy, Microsoft, USA and Dr. Jacques Bulchand-Gidumal, 
Professor of Digital Enterprises and Entrepreneurship, Universidad de Las Palmas de 
Gran Canaria, Spain as the e-Learning 2015 keynote speakers. 

A successful conference requires the effort of many individuals. We would like to tha nk 
the members of the Program Committee for their hard work in reviewing and selecting 
the papers that appear in this book. We are especially grateful to the authors who 
submitted their papers to this conference and to the presenters who provided the 
substance of the meeting. We wish to thank all members of our organizing committee. 


xi 



Last but not the least, we hope that everybody will have a good time in Las Palmas de Gran 
Canaria, and we invite all participants for the next edition. 


Miguel Baptista Nunes, University of Sheffield, United Kingdom 
Maggie McPherson, University of Leeds, United Kingdom 
e-Learning 2015 Conference Program Co-Chairs 

Piet Kommers, University of Twente, The Netherlands 
Pedro Isaias, Universidade Aberta (Portuguese Open University), Portugal 
Heredina Fernandez Betancort, Universidad de Las Palmas de Gran Canaria, Spain 
MCCSIS 2015 General Conference Co-Chairs 
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KEYNOTE LECTURES 


“FAIL FAST AND FAIL FORWARD - EMBRACING 
FAILURE AS A NECESSARY PRECURSOR OF SUCCESS 
IN THE DELIVERY OF ELEARNING SERVICES” 

Steven Duggan, 

Director, Worldwide Education Strategy, Microsoft, USA 


ABSTRACT 

How cloud services, low-cost mobile devices and real-time data can enable system design and 
course correction in national deployments. 

With reference to eLearning partnerships with organizations including UNESCO, World Vision 
and British Council, Steven Duggan, Microsoft Director of Worldwide Education Strategy, will 
outline how a shift in the culture and approach to Public Private Partnerships is required to 
achieve outcomes which can successfully be replicated at scale. 

With a focus on the global challenge of illiteracy, Duggan will explain why an openness to 
recognize and value failure proved essential to the building of new solutions which could 
address access, equity and quality in the delivery of eLearning services. 


A DIFFERENT PERSPECTIVE ON THE SINGULARITY 
POINT. HOW IT IS SUBSTITUTING JOBS IN THE 

SERVICE SECTOR 

Dr. Jacques Bulchand-Gidumal, 

Professor of Digital Enterprises and Entrepreneurship, 

Universidad de Las Palmas de Gran Canaria, Spain 


ABSTRACT 

While the Singularity Point is announced to take place around 2030, before it happens humans 
will have to face several challenges. Among them, a more than probable substitution of a large 
amount of jobs by IT. While for some time computers and robots seemed to be especially well 
oriented to substituting industrial jobs, latest developments have contributed to picture an 
situation in which many service sector jobs are also being replaced. We posit that this maybe 
one of the causes behind the global crisis that is taking place right now and of why it is hitting 
so much in service-based economies. Some future realistic scenarios are explored. 
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2 INDIRE - Istituto Nazionale di Documentazione , Innovazione e Ricerca Educativa, Sede territoriale Nord 

Corso Vittorio Emanuele II, 70 - 10121 Torino 


ABSTRACT 

In this work we will introduce the concept of music coding, namely a new discipline that employs basic music activities 
and simplified languages to teach the computational way of thinking to musically-untrained children who attend the 
primary school. In this context, music represents both a mean and a goal: in fact, from one side coding activities are based 
on music processes that are able to unveil algorithmic thinking; from the other side, such processes may stimulate 
creativity and collaborative learning, and their audio feedback is immediately perceivable. Consequently, music can 
represent a valid learning tool as well as an addictive reinforcement technique to approach coding. After describing a new 
formalism to graphically encode music operators, a Web prototype specifically designed for music coding will be 
presented and discussed. 


KEYWORDS 

Music, coding, web. application, primary school. 


1. INTRODUCTION 

Media and technology are nowadays characterized by a network system widely structured which encourages 
the development of phenomena with important consequences both in social and educational fields. Among 
them a prominent role is played by so-called digital convergence, i.e. the process by which different media 
are redefined in their peculiarities thanks to digital coding. This cannot be merely seen as a technical or 
technological development, as this process has changed irreversibly media industry and the way media 
products are enjoyed. 

Within a convergent culture, meanings - even music meanings - circulate according unexpected routes, 
contaminate each other, undergo discontinuous processes of certification, finding only temporary 
consolidation and being subjected to constant revisions (Ardizzone et ah, 2008). In these complex and 
heterogeneous universes, sound and music take a new and stronger value as languages integrated into broader 
and diversified contexts, each leading to a hyper-fruition of aural stimuli. 

It is worth underlining another key aspect, represented by the new technological opportunities of 
interaction, with the overcoming of the idea of audience proper of the Gutenberg model and the development 
prosumers increasingly active (McLuhan & Nevitt, 1974). Through digital technologies and the great impulse 
of Web the sharp dichotomy between producers and consumers disappears, similarly to what happened to 
communication processes in the 90’ (Ceri & Gallino, 1994). 

Due to the deep and rapid changes in the world around us, occurring also through media and technologies, 
the concept of literacy has to be rethought by encompassing a multiplicity of languages. Some researchers 
state that, in order to achieve socio-educational integration, it is not possible to conceive models of 
educational relationship other than inclusion, multi -literacy and promotion of excellence (Napodano & 
Iandoli, 2008). This is a set of educational actions that enhance workshop activities, foster the ability to use 
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language, orality and heterogeneous systems for analog and digital encoding, and promote the variability of 
cognitive styles and communication registers according to Gardner’s idea of forma mentis. 

New challenges emerge in modern school, traditionally considered as the depositary of literacy processes: 
guaranteeing the growth of students, each one with his/her own peculiarities, and giving equal dignity to the 
languages they interact with in everyday life. As regards music language, the problem is not how to employ 
media technologies during traditional hours of music education, but completely rethink their value in order to 
train individuals along different axes: searching in sound and timbre collections, analyzing their role and 
meaning inside a complex media message or a soundscape, understanding the value of music and audio 
authorship, and much more. 

For an actual acquisition of competence, intended here as the ability to act in - or react to - a given 
situation within a given context in order to achieve a performance (Le Boterf, 1994), it is necessary to 
develop education proposals able to integrate music analysis with music production. In this sense, coding can 
prove to be an effective proposal since it implies writing, namely making explicit, juxtaposing and processing 
concepts in a critical, aware and creative way. 

The paper is organized as follows: Section 2 discusses the pedagogical aspects of coding in primary 
school curricula. Section 3 illustrates the relationship between music and coding starting both from the music 
and from the technological points of view. Section 4 presents a Web interface that implements music coding, 
and finally Section 5 proposes a number of case studies. 


2. FROM CODING TO MUSIC CODING 

In Computer Science, the term coding refers to the implementation of a program, namely a sequence of 
instructions that make a computer run a given set of actions. It is well known that computer programming is a 
base skill required to understand information technology processes. More surprisingly, in recent times coding 
has been proposed also as a tool to foster learning processes in young students. By facing coding problems, 
students activate many cognitive functions that go beyond the acquisition of technical expertise in a strict 
sense. Coding tasks should be independent from a specific programming language or software environment. 
Rather, the target is developing a digital competence through playful learning (Resnick, 2004). 

The theoretical bases go back to the pedagogical approach of constructivism (Piaget et al., 2014). As 
regards the introduction of coding and computational thinking in primary school, it is worth citing the ideas 
of Papert: while each discipline claims to push students to think, computer science achieves this result in an 
operative and concrete way. Papert considers coding as a useful tool to learn how to think, not as a way to 
obtain improved reasoning skills. In fact, the intrinsic goal is not making children good programmers, but 
providing the knowledge to create contexts where they can explore advanced ideas (Papert, 1980). For 
example, young learners can approach differential geometry through the metaphor of a turtle or they can 
understand the concept of feedback thanks to LEGO robots. Papert adopts LOGO as the reference 
programming language for children’s coding activities. An implementation of Papert’s ideas requires a deep 
change in teaching, transforming the mere transmission of knowledge and skills into laboratory activities and 
structured projects aimed at encouraging collaboration and discussion. 

Music education and training for children require ad hoc techniques and methods as well as a specific 
review of school curricula. Recent scientific works (Young, 2003) show that an integration of multi-modal 
experiences based on activities such as moving, creating, playing, reflecting support the development of a 
“symbolically fluent child” (Gromko, 1995). The learning environment should be able to represent 
activity-oriented musical experiences, where students - properly sustained by scaffold elements - are 
involved in a process of music construction/deconstruction. For example, scaffolding children’s early musical 
experiences, investigations and engagement in the world of sound helps them establish strong, confident, 
vibrant, and creative identities in learning, communication, and performance (Tomlinson, 2013). 

Currently a new music pedagogy based on an integrated approach is emerging. Recalling the fundamental 
concepts of pedagogical activism (Dewey, 2005), the goal is enhancing that educational cross-component 
able to influence key aspects of the growth such as expressiveness, autonomy and sociality. 

Music is able to influence the construction of the child’s personality because it promotes the integration 
of perceptual, motor, affective, social and cognitive dimensions (Willems, 2011) by relating basic aspects of 
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human life (e.g. physiological, emotional and mental spheres) with the basic elements of music (e.g. rhythm, 
melody and harmony). 

The abilities of listening, exploration and analysis are fundamental for the development of general 
meta-cognitive skills of the child, such as attention, concentration, control. In this sense, music is both an 
opportunity and a crucial educational strategy. For example, through music young students can develop the 
aspects of analysis and synthesis, problematization, argumentation, evaluation and application of rules. In 
addition, as regards the ability to read and understand, children have the possibility to train their transcoding 
skills, moving from the musical domain to the verbal language in order to describe what they heard (Branca, 
2012 ). 

In the digital era, new technologies and computer-based approaches can influence music learning and 
teaching processes. A recent and comprehensive review of this subject can be found in (Finney & Burnard, 
2010), a work that discusses a range of innovative practices in order to highlight the changing nature of 
schooling and the transformation of music education. Many researchers, experts and music teachers feel a 
pressing need to provide new ways of thinking about the application of music and technology in schools. It is 
necessary to explore teaching strategies and approaches able to stimulate different forms of musical 
experience, meaningful engagement, creativity, teacher -learner interactions, and so on. 

The idea of the this work is applying the most recent pedagogical theories about coding and music 
teaching in primary school through a playful approach to music composition, conceived for musically 
untrained children and designed to encourage the computational way of thinking. 


3. MUSIC OPERATORS AND COMPUTER PROGRAMMING 

In this section we want to explore the historical, cultural, and theoretical relationships between two domains 
apparently far apart, i.e. music and computer programming. We will follow two approaches somehow 
complementary, highlighting how some concepts of music can find a counterpart in the constructs of 
programming languages, and how some concepts typical of algorithmic way of thinking can be intuitively 
represented through music operators. 

3.1 From Music to Coding 

In 1985 composer and music theorist Fred Lerdahl and linguist Ray Jackendoff conceived a generative 
theory of tonal music (Lerdahl & Jackendoff, 1985). They provided a “formal description of the musical 
intuitions of a listener who is experienced in a musical idiom” based on the unique human capacity for 
musical understanding. The purpose was uncovering a musical grammar that could explain the human 
musical mind in a scientific manner comparable to Noam Chomsky’s transformational or generative 
grammar dating back to 1975 (Chomsky, 2002). The novelty of their approach was the investigation of the 
mental procedures under which the structure of individual compositions can be built and understood. 

Even if the approach of the mentioned authors was original, many theorists, musicologists and composers 
in the past formalized or merely used algorithmic processes to conceive music. Examples range from 
late-Baroque complex counterpoint processes (see e.g. the fugues by Johann Sebastian Bach) to twelve-tone 
serialism (see e.g. the works by the Second Viennese School, composed by Alban Berg, Anton Webern, 
Hanns Eisler and Arnold Schoenberg). As regards theoretical works, it is worth citing (Hofstadter, 1980), 
where the concepts of self-reference and formal rules are applied to the domain of mathematics, art, and 
music, and (Bizzi, 1982), which unveils the way many historical canons have been composed through 
mathematical rules and tables. 

The mentioned works demonstrate that a formal and algorithmic approach to music composition and 
analysis is possible. Consequently, we introduce a number of music operators that can be seen as single steps 
of a generative algorithm. Due to the goals of this initiative, the application domain is simplified: 

• A score is composed by a number of melodies, one per staff, presenting no chords. Harmony-related 
aspects come from the coexistence of notes belonging to different instruments and staves; 

• Every note is quantized according to the smallest rhythmical value allowed. Longer values can be 
obtained by tying quanta together. This approach is valid for children's songs and simple tunes, where a 
coarse quantization is sufficient, but it would fail with tuplets and more complex rhythmic layouts; 


5 



ISBN: 978-989-8533-40-1 ©2015 


• Supported operators are simple to be used and understood. The goal is not covering all possible 
music-composition processes, but provide untrained children with an intuitive tool set to build basic 
music performances. 

3.1.1 Melodic Operators 

The first class of music operators is the one of Melodic Operators (MOs), namely the operators that work on 
note pitch. The proposed framework includes: 

• Play n - This operator starts playing a n-pitched note. The execution is stopped at the end of the current 
quantum, unless a Continue operator (see Rhythmic Operators below) is invoked at the next quantum; 

• Clear - This operator unsets the pitch of the current quantum. In a quantized environment, each note 
naturally takes a single quantum, nevertheless a quantum cannot be empty. In this sense. Clear is the 
default operator, and it corresponds to the music concept of rest. This behavior implies that Clear 
belongs both to Melodic Operators (since it unsets pitches) and to Rhythmic Operators (since it is a 
placeholder for a rest); 

• Transpose up/down ( and play) - This operator modifies the previous pitch according to a number of 
ascending or descending grades. In our implementation, we decided to deal with scale grades (referring 
to a diatonic scale) rather than halftones (referring to a chromatic scale) because most pitched 
instruments for children are mainly diatonic, such as toy xylophones and whistles. 

3.1.2 Rhythmic Operators 

Rhythmic Operators (ROs) work on rhythm -related aspects of notes. In this case, the framework includes: 

• Tie - This operator extends the last note (or rest) for the duration of the current quantum; 

• Clear - As mentioned above, this operator means absence of sound. So, even if a note is naturally 
stopped by the end of its quantum. Clear has rhythmic implications too. 

3.1.3 Other Operators 

In order to foster computational thinking, other operators which do not belong to the mentioned classes have 
been implemented. A first case is Repeat, which allows to choose a number of previous steps to be repeated 
for a given number of times. In music notation there are symbols that recall this function (e.g., n-bar repeat 
signs, repeat barlines and text indications such as “Da capo”). However, for the sake of clarity the 
implemented Repeat operator has a less flexible applicability. 

On the contrary, conditional expressions and constructs do not have a music definition, but they have 
been introduced to foster algorithmic thinking (see Section 3.2). In the proposed framework, comparison 
operators such as lower than, higher than, lower or equal to, higher or equal to, equal to, and different from 
can be invoked on pitches. 

3.2 From Coding to Music 

In Section 3.1 we have listed a number of music operators that are implicitly or explicitly used by composers 
when they create a music piece (Schoenberg & Stein, 1970). We have shown how these operators can be 
converted into atomic steps of an algorithm. On the contrary, in this section we start from some key concepts 
typical of computer programming and try to map them onto the music case study. 

In our approach, programming constructs intentionally are not presented in an explicit way to children. In 
other words, the proposed framework hides their presence, but makes the underlying concepts emerge. 

First, let us consider the standard structure of a program. Most programs have the following pattern: i) 
statements to mark the start of the program, ii) variable declaration, and iii) program statements. This basic 
schema can be mapped on: i) choice of the basic parameters (tempo, score length) and creation of 
instruments, ii) choice of the instruments to play, and iii) for each instrument, selection of a set of actions to 
be performed. Code lines are usually written and read from top to bottom, whereas music code - in 
accordance with score notation - should be interpreted from left to right. 

In computer programming, data typing implies a classification identifying one of various types of data 
(e.g. integer, real, boolean, etc.) and defines allowed values and available operations on such values. In our 
proposal numeric, character and boolean values can be selected to perform specific operations, such as 
setting pitches or testing conditions. However, the novel approach is mapping the concept of data type onto 
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the choice of an instrument type to play. If instruments can be seen as instances of variables, then instrument 
types correspond to data types. For the sake of clarity, we will present the case of a xylophone vs a drum, 
which are two of the available instruments supported by the interface described in the following. Please note 
that they are both percussion instruments, but the former is pitched whereas the latter unpitched. The 
xylophone allows to define the exact note to be played. This implies the availability of operators such as 
Transpose up or Compare pitches. Besides, in accordance with computer programming where values are 
limited to a range (e.g. in C integer values have to fall in the [-32768,+32767] range, being 16 bits in size), 
pitch values present limits that depend on the chosen instrument. Needless to say, for an unpitched percussion 
instrument such as a drum the concept of pitch has no meaning, consequently only rhythmic operators are 
supported. 

The concept of data typing can be further detailed. In computer programming, there is a dichotomy about 
strong vs weak typing. In our proposal, this aspect can be found in the independent use of instruments against 
the interdependence of notes across different instruments. 

As regards operations on variables, assignment can be employed at different stages. For example, 
deciding which actions should take place at a given time is a form of assignment. Besides, some of the 
supported operators allow to set additional values, such as the pitch to be played or the number of repetitions 
to be performed. A complete list of operators for this proposal will be provided in the following. 

As regards basic control flow , most programming languages support unconditional branch (e.g. goto, 
break, return, etc.), conditional branch (e.g. if-then-else, switch, etc.) and loops (e.g. while-do, do- while, for, 
etc.). In order to apply conditional statements to the music case, we introduce comparison operators to check 
for conditions and consequently choose between two alternative execution paths. For example: “If the current 
note of Instrument A is lower than the current note of Instrument B, then follow the upper path, otherwise 
choose the lower one. In order to keep things simple, braches always present the else clause and have the 
same length. 

Loops and iterative constructs are exemplified through the repetition of the last n steps for k times. Please 
note that these statements introduce a number of implementation issues that in standard coding activities do 
not arise. The last n steps to be repeated do not have to generate ambiguous or illegal situations, pointing for 
example to a step where different branches are present (at the moment of coding, the system is not aware of 
conditional choices which may depend on run-time decisions). Similarly, a standard while-do loop would be 
difficult to introduce since the exit condition - and consequently the loop duration - cannot be predicted. 

Some non-trivial programming features are supported too. First, data types are not limited to simple ones. 
The concept of array can be intuitively represented by a particular set of instruments (an array of mixed 
variables). If we define the array K by grouping a subset of instruments, operators can refer to single-element 
as well as aggregated values. For example, in conditional statements we can check conditions such as: “If the 
current note is higher than all current notes in K...” or “If current note of Instrument K t is higher than the one 
played by K 3 ...". In this way, children can get familiar with the access to array elements by index (e.g. K[1 j) 
or by key (e.g. K[“drum ”]). 

Finally, the specific implementation of some operators recalls advanced concepts such as overloading, 
namely a type of polymorphism where different functions with the same name are invoked based on the data 
types or the parameters passed. For instance, the operator Play can be invoked with no parameters - i.e. 
Play(), which repeats the last pitch if present, and performs the default one otherwise - or with one parameter 
- i.e. Play(n), which explicitly sets the pitch to be performed. Similarly, Repeat() means “play the last music 
symbol once again”, Repeat(x) means “play last x music symbols once again”, and Repeatf x,y) means “play 
last x music symbols y times”. These are simple but effective examples of polymorphism. 


4. WEB PROTOTYPE 

In order to make music coding appealing, amusing and accessible to children, we have implemented an 
experimental Web interface adopting a gamification approach. The use of standard Web technologies and 
languages such as HTML5 and Javascript, developed and promulgated by W3C, 1 let us release a 
cross-platform multimedia environment, available on all browser-equipped and network-attached devices. 
The interface is shown in Figure 1. 


1 World Wide Web Consortium 
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Figure 1. The Web interface designed for music coding. 


4.1 The Gameplay 

The educational activity is composed by two different steps, named score coding and interactive listening 
respectively. 

During the first phase, a score is produced by disposing a number of cards (corresponding to music 
operators) on the game board. Available operators are chosen by clicking the instrument name, which opens a 
contextual toolset, since different instruments may present different sets of operators. The score length is not 
fixed, on the contrary it is set on-the-fly while dragging cards. The available space in the interface can be 
extended thanks to horizontal scrolling. As explained below, during this process the corresponding 
“traditional” music score is produced as well. The coding process can be performed collaboratively by many 
children together, for instance by assigning to each student either an instrument or a step, like in turn-based 
games. 

The interactive listening phase starts when the Play button is pushed causing playback to occur. When 
score steps are parsed, colored balls begin to jump over the instrument interfaces. Graphical and audio 
materials are fully synchronized, thus allowing a multi-layer score-following experience. For children it is 
possible to look at the ball moves, follow music over the graphical as well as the traditional score, and listen 
to the corresponding sounds. This process somehow resembles debugging, since actions are parsed one by 
one to see if the desired result is being achieved. Music performance can be paused, stopped, and rewound. 

The second phase has been defined interactive for a number of reasons. First, music performance can be 
influenced by adjusting and rearranging music operators: at any moment users can modify the score, thus 
going back to phase 1. Second, children can passively enjoy the results of coding but they can also interact by 
playing their own instruments. 
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4.2 The Interface 


As regards the interface, the main area of the screen contains a number of pitched and unpitched music 
instruments. The available orchestra can be configured during the game setup. For music playback, the 
underlying metaphor is a jumping ball that strikes the sensitive areas of each instrument to make it play. 
Needless to say, jumps are controlled by the score. 

The upper panel represents the game board, namely the area where cards corresponding to music 
operators are placed. Rounded squares constitute the quantized timeline, and their colors refer to a given 
music instrument. Icons like those shown in Figure 2 are employed to confer an intuitive graphical 
representation to music operators. 



Figure 2. The icons for Play, Transpose down. Transpose up. Tie, Repeat and Clear music operators. 

The bottom panel contains a score transcription following Common Western Notation (CWN) rules. 
Music contents are automatically generated over the staff system while cards are being placed on top of the 
game board. This “traditional” score representation has not been conceived for children, nevertheless, its 
presence can be important for a number of reasons: 

• It can be checked by musically-trained students and teachers to validate the use of music operators; 

• It can be directly read by young musician to create a live performance or an accompaniment; 

• Thanks to full synchronization among all graphical and audio contents ( score following), it can be 
an intuitive way to learn the rudiments of music notation. 

An inverse exercise based on score notation is trying to reconstruct a simple tune whose score is known 
through the operators provided by the game. In this case, coding is involved as a problem-solving technique, 
similarly to standard computer programming: given a goal, find the self-contained step-by-step set of 
operations to be performed in order to achieve it. 


5. EXAMPLES 

Now let us refer again to the screenshot shown in Figure 1 . This time we will focus on cards over the game 
board and the corresponding music contents. 

The upper staff refers to a pitched instrument, consequently all music operators are available. From the 
point of view of rhythm, no steps but the last one have been extended through ad hoc operators. The first 4 
steps are explicit assignments indicating the pitch to be performed. The 5th step is an overloaded version of 
the assignment operator, presenting no indication about the pitch: in this case, the last pitch is repeated. Other 
in use operators are simple transpositions (1 grade up or down) and simple repetitions, i.e. single repetitions 
of the last step. Clearly the same result could be achieved through different combinations of operators: for 
instance, the 3rd and the 4th step could be obtained through transpositions. This is another aspect of music 
coding close to computer programming, where different algorithms can be employed to achieve the same 
result. 

The second and third staves encode parts to be played by unpitched instruments. For an untrained child, 
starting from a simplified set of operators can be more intuitive. Besides, a rhythmic feedback is easier to 
recognize. 

In this example, we have used a number of Repeat operators, thus originating aggregations of steps 
different in length and number of repetitions. The notation at the 3rd step of the bottom staff is particularly 
efficient in representing a huge area of the score. In computer terminology, we would call it compression. 


6. CONCLUSION 

The concept of music coding presented in this work constitutes a novel approach to the computational way of 
thinking. Actually, playing music is a stimulating and amusing activity for children, which fosters not only 
technical and theoretical abilities, but also creativity, adaptivity, socialization and cooperation with others. 
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The interface and the underlying language presented above address primary school children without a 
specific music background. They have been designed to allow easy interaction with music contents as well as 
intuitive manipulation through ad hoc operators. Nevertheless, young musicians and music teachers can be 
involved in these activities as well. From a passive perspective, they can enjoy an alternative and formal 
representation of music language, quite far from common notation but close to the composer’s way of 
thinking. From an active perspective, musically-trained users can join the playback phase, acting the role of 
additional instrumentalists and arrangers. 

It is worth underlining that the provided framework can be seen as a tool both for music synthesis 
(“choose cards, set parameters and listen to the music produced”) and for music analysis (“analyze an already 
available tune and try to reconstruct it through a suitable combination of cards and parameters”). 

At the moment of writing, the software application has been implemented and is publicly available over 
the Web, but it has not been extensively tested. As regards future work, a relevant goal is to conduct a test 
phase in Italian primary schools, in order to understand user acceptance rates and to validate the pedagogical 
approach. For instance, we are planning assessment activities in order to test if a music-based approach to 
problem solving can be more effective than a traditional one, by proposing the same algorithmic problem in 
different forms. An example is reaching a target point by walking in an open space versus composing a 
simple tune, two clearly different activities that could be described through the same sequence of algorithmic 
steps. Another kind of exercise is recognizing algorithmic structures in a given music piece. 

A number of open problems must be faced before the final release of an effective learning system. These 
problems suggest a variety of research directions in the fields of pedagogy, technology-aided teaching and 
learning, computer science, music, and musicology. 
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ABSTRACT 

Distributed learning is a growing issue in education following the mainstreaming of technologies such as 
videoconferencing. However, though distance and distributed learning have been common in adult education and 
business since the 1990s little is still known about the use of videoconferencing in elementary education. This paper 
reports from ongoing research in three rural schools in Denmark where the use of videoconferences are used as part of the 
teaching at lower secondary level. The research focuses on how students learn from videoconferences that are both one- 
to-many and peer-to-peer. Videoconferencing, conceptualized by the schools in question as telepresence, is performed in 
a unique combination of desktop interaction through mobile devices (iPads) and studio-based large screen lectures and 
interaction. Preliminary results of the research suggest that telepresence could be enhanced in schools by seeing it as a 
broad framework for collaboration between schools, in which different kinds of connections - both synchronous and 
asynchronous, mediated and face-to-face, large screen and desktop interaction - can support the placement of schools 
within the community and in the global context of learning. 


KEYWORDS 

Telepresence, learning through videoconferences, learning with iPads 


1. INTRODUCTION 

In the spring of 2013 I was asked to research an ongoing school development project in a rural part of the 
west of Denmark, where three schools had for a year been using videoconferencing for teaching basic 
subjects such as maths, German, science, history and literature. The purpose of the project, which was 
initiated by the local municipality, was to explore how the use of videoconferences in schools could support 
community and curriculum developments through increased collaboration between schools, access to experts 
and the establishment of links between schools, industry and community. Within these broad aims the 
municipality sought to respond to a variety of community challenges, for instance the depopulation of 
specific rural areas and the consequent limited access to resources such as specialized teachers and experts. 
Following a municipal reform in 2007, the municipality had increased in size and incorporated several rural 
school districts which necessitated a local strategy for linking schools and reorganizing available resources. 
As a result, the municipality initiated the project and at the same time schools invested in iPads for all 
students in lower secondary school (age 13-16). 

The research project specifically explored the potential of videoconferences as a supplement to classroom 
teaching for qualifying lower secondary students’ learning, with the three local schools as an empirical case. 
Research was done as an ethnographic, multi-sited study that included observations in all three schools as 
well as interviews with students. Both telepresence activities and classroom activities prior to and after the 
telepresence activities were observed, whenever possible. Research is ongoing and the following reports from 
the initial phase of the project from the autumn 2013 to 2014. The project ends in the summer of 2015. 
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2. UNDERSTANDING TELEPRESENCE: INSIGHTS FROM THE 
LITERATURE 

Telepresence is a broad and complex term for a variety of phenomena involving embodied, virtual presence 
in mediated environments (Friesen 2014, Levinsen et al. 2012). From a phenomenological perspective 
telepresence can be understood as “the experience of presence in an environment by means of a 
communication medium” (Steuer 1995 in Dolezal 2009) or conversely as “a technology that allows the 
projection of a person to a remote environment” (Draper 1998). As a concept, telepresence has historically 
been linked to the use of robotics and virtual worlds, however, telepresence also interacts significantly with 
local physical spaces, as will be seen from the examples below. 

In this paper I shall focus on videoconferences as a telepresence activity, where telepresence is 
understood as a way of configuring learning rather than a specific experience of being present. In 
understanding telepresence through videoconferences as a process of configuration, I am inspired by the idea 
that telepresence is contingent on local practices of for instance collaboration, subject cultures and modes of 
instruction (Pink 2011). In this sense telepresence is placed in local practices and entangled within these 
practices (Dussel 2013, Latour 2005), rather than in individual learners’ experience of presence. Researching 
telepresence therefore involves studying new ways of placing technologies in schools, as well as new ways of 
placing schools in the community. 

According to the literature, videoconferencing can be defined as “synchronous audio and video 
communication through computer or telephone networks between two or more geographically dispersed 
sites” (Cole, Ray, & Zanetis, 2004 in Lawson 2010). The potential of synchronous interaction is immediate 
interaction with others distributed in geographical space, which may extend learners’ access to high-quality 
learning opportunities (Berge & Clark 2005, Barbour & Reeves 2009). As the practice of learning through 
videoconferences originated in higher education and business contexts (Lawson 2010, Falloon 2012) the role 
of videoconferences in learning are still relatively unexplored in elementary schooling, though successful 
experiences have been made in rural schools and with home-schooled children (Barbour & Reeves 2009, 
Lawson 2010). Barbour and Reeves (2009) suggest that the use of videoconferences for learning may be 
difficult to translate into elementary school contexts, as successful learner characteristics such as autonomy 
and intrinsic motivation, which are usually associated with adult learning, are still active in these 
environments. Introducing videoconferences in lower secondary schooling will therefore require a 
reconceptualization of traditional pedagogies. 

Though videoconferences, according to the literature, have the potential to increase social presence, 
community and educational opportunities in specific local areas and for specific students, several challenges 
are nonetheless connected with establishing these new social arenas. Challenges may include collaboration 
issues such as time-table coordination, booking requirements and technical limitations (Pitcher et al. 2000, 
Lawson 2010). In addition to this, modes of instruction such as one-to-many lecture formats may disengage 
young learners who are used to more enquiry based learning. In the telepresence with iPads project schools 
sought to meet this challenge by constituting telepresence within the unique combination of studio-based 
lecture-like activities and student-to-student interaction through personalized technologies (iPads). Within 
this conceptual matrix, telepresence unfolded as specific configurations of videoconference activities that 
were meant to suit learners in the 7-9 th grade. 


3. MOVING BETWEEN SITES - METHODOLOGICAL ISSUES 

Because of its focus on distributed learning, the telepresence with iPads research project has been inspired by 
multi-sited ethnography (Marcus 1995, Hannerz 2003). According to Marcus and others, multi-sited 
ethnography has provided a methodological framework for ethnographically following things, ideas and 
people in global contexts where phenomena are mobile and transient. Multi -sited ethnography thus moves 
away from the single-sited practice of conventional ethnography and follows the circulation of objects, 
practices and identities across sites. 

In the telepresence with iPads project multi-sited ethnography has contributed to mapping relationships 
between schools and studying their collaboration through telepresence. As collaboration between schools is 
built into the project, multi-sited ethnography has followed telepresence as a phenomenon that emerges 
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within these collaborations, i.e, the idea is that looking at it from one locality is only seeing it partially. In the 
project I as the researcher for instance had to map and travel distances between schools in order to understand 
how distance and other kinds of (dis)connections might act in relationships between local schools. In 
mapping and travelling distances between schools, I had to decide, in relation to each telepresence activity, 
where to be physically present and how to incorporate in my observations the experiences and learning of 
those that were remote. As the research was multi-sited, I would, if possible, try to shift between sites where 
the teacher or instructor was remote and where she/he was physically co-present with me in the classroom. In 
addition to this, I would, if possible, visit the collaborating school(s) later the same day or the day after to 
have informal talks with teachers or do interviews with the students. However, because of the organizational 
constraints mentioned below, this rhythm of presence was not always possible. 

As physical presence in several sites was a prerequisite for understanding telepresence in the project as 
seen through the multi-sited lens, researching telepresence ethnographically paradoxically entailed travelling 
the distance that videoconferencing attempted to bridge. Latour argues that “. . . elements which are close 
when disconnected may be infinitely remote if their connections are analyzed; conversely, elements which 
would appear as infinitely distant may be close when their connections are brought back into the picture.” 
( 1996, 4). As suggested by Latour, research has to identify the complexities of these connections which are 
not only geographical, and my research therefore had to map out the multitude of ways in which connections 
and their distance-closeness effects could affect students’ learning. What this entailed was to some extent that 
telepresence as a phenomenon was created through my trajectories of observation and presence and the 
connections I was able to make between them. 

In connection with this, my role as a researcher was generally affected by the complexities of being an 
observer between several sites, including those that were connected with my position as a university 
employee. As an associate professor employed by a state university I was situated in Copenhagen, a 3-4 
hours’ drive from the local area I was researching. As someone belonging to a different community, my 
distance from the field often seemed more extended and remote even when I was physically present in the 
schools. This became increasingly obvious as I struggled to maintain contact with the schools through email 
and phone calls, in order to be included in telepresence events. The time/space displacement of researcher 
and field acted through my role as researcher, as telepresence events where often not planned ahead, but 
incorporated into local practices and time rhythms. As I was conceptualized as a remote actor with no 
immediate connections to practice I was often excluded, though I generally felt very welcome in the schools. 

Finally, distance and proximity acted in through the relationships between schools and their affiliations 
with the local community. All three schools, though in average 15-20 kilometers distant from each other, 
were as mentioned above situated in the same municipality, and in many ways shared local histories and 
socio-economic conditions. However, organizational differences between the schools, such as differences in 
time tables, teaching styles and practices, often challenged the realization of collaborative telepresence 
events. In addition to this, teachers and students, though all in one perspective local, were mostly strangers to 
each other, and had not generally been given time to meet, get acquainted and plan ahead. For many teachers, 
collaboration is in itself a challenge as they are used to managing classrooms on their own, and opening up 
the classroom through videoconferencing can be daunting. In this sense local proximity - as argued by the 
literature - needs additional affiliations to create trust and give access to classrooms (see for instance Friesen 
2014). In fact, in one case collaboration between two teachers was only initiated, I was told, because the 
teacher in school A knew the teacher in school N as a parent, i.e. his son attended school N where the 
collaborating teacher taught. What seems to be close in terms of local placement may therefore not be 
sufficient to support the relationships between schools that are necessary for collaboration. 


4. CASE 1: GEOGRAPHY THROUGH TELEPRESENCE. 

School A is a small, rural school in the west of Denmark where the students have been working with 
telepresence through their iPads as well as large screens situated in the classroom dedicated to telepresence 1 . 
The students are working on a project in geography with school N, a larger, rural school situated about 16 
kilometers away. The students in school A and school N have not met each other physically before the 


1 School A has a population of 214 students. School N 294 students, school T 257 students 
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telepresence session, however, a few of the students know each other from communion class, leisure 
activities or other contexts. 

School A and school N are collaborating on a geography project where Google Earth is used to tag places 
in different countries. According to the teacher in school A, the point of the project is to raise the students’ 
awareness of local differences in countries like for instance Norway, USA, Belgium and Holland. Through 
Google Earth, he explains, students will ideally be able to identify infrastructural differences through the 
visual representations of local spaces in the Google Earth interface. As the project is collaborative between 
the schools, the tags are swapped in a kind of treasure hunt for national, geographical differences. 

In the lesson I am observing, videoconferencing is used to instruct students in both schools in how to 
place the tags in Google Earth used for the treasure hunt. The teacher in school N is teaching the students in 
both schools, as the teachers have divided the tasks between them, and he is the only one who knows the 
functionalities of Google Earth. This means that the students in school A are obliged to address the remote 
teacher if they need help in understanding the assignment. 

The focus of the telepresence session is, as described above, the identification and study of local/global 
space as a geographical phenomenon. Thus, images of the earth are projected and navigated easily by the 
remote teacher on the large screens covering the end of the wall that faces the students (see figure 1). In 
quick successions, students’ gaze is invited to travel from a view of the Earth from out of space to a view of 
their local area in Denmark and then further into European locations. As shown in the photo below, students 
are meant to follow and replicate the remote teachers’ instructions on their own pcs and iPads - which 
potentially adds a feeling of proximity to what is illustrated as well as draws on a student centered approach 
to the instruction. However, as I shall argue below, the instruction generally fails to engage the students in 
school A, who for the most part very quickly become completely disengaged from the activity. 



Figure 1. Geography through videoconference: school A and school N 


Location and geographical space is thematized by the geography lesson as described, however, as 
illustrated by figure 1 , issues of location are active not only as an aspect of what is taught, but as part of the 
learning situation itself, i.e. as modes of proximity and remoteness in the telepresence activity act actively in 
the learning of the students. This echoes the argument made by Richardson & Wilkin (2012), that the 
supposedly dematerializing effects of cyberspace and telepresence - in this case the projections on the 
screens - is always unfolded within specific, localized contexts and activities. In the case at hand, schools are 
localized within a distance of less than 20 kilometers from each other and belong to the same municipality, 
which in some ways - for instance in terms of community - seem integrated and close. However, distances 
between the students and teachers in school A and N are nonetheless not easily negotiated in the specific 
session, as students in school A become increasingly passive during the geography lesson. What acts in the 
situation and inspires (dis)engagement is, it seems, not geographical proximity or local affinity, but modes of 
attachment and instruction that keep the students from actively participating in the lesson. Thus, as the 
students are not familiar with the remote teacher, they fail to address him when they do not follow the 
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instructions on the screen. As a rule, they prefer to ask their local teacher for help, but, because of the 
division of labor between the teachers, the teacher at school A cannot help them with the task. In addition to 
this, the mode of instruction and possibly the way in which school N is presented on the screen - displayed 
from a distance and with no eye contact with school A - disengages students from both the partner school 
and the subject at hand. What seems to be near in terms of geographical proximity and local identity 
therefore presents itself as infinitely remote in the specific context of the geography lesson observed. In fact, 
as the screens simultaneously show an out of space perspective on the earth and a view of the remote teacher 
standing at the front of his class in school N, these spaces seem to be at the same level and at a similar 
distance. This is in spite of the intruding soundscape that all through the session penetrates the relative quiet 
of the classroom in school A with the shrill voices of students of school N. In this way the community may 
be brought into the classroom but not engaged with in significant ways. 


5. CASE 2: LEARNING GERMAN THROUGH TELEPRESENCE 

Case 2 consists of two separate scenarios where I observed how German was taught through telepresence in 
the same class in two different school years in school T, in collaboration with respectively school A and 
school N. In the first scenario German was taught peer-to-peer, i.e. through iPads, with FaceTime interaction 
between students in school T and school A. The students, who were at that time beginners in German as a 
foreign language, were asked to take turns with presenting themselves in German to the students in the 
collaborating school. This was done with the support of a piece of paper with notes that they had prepared 
before the lesson. In the second scenario students in school T and N were taught German adverbs one-to- 
many by a teacher at school N, who taught the basic grammatical principles of adverbs through a PowerPoint 
in a videoconference on the large screen. After the presentation, students at school T and N were asked to 
translate sentences from Danish into German that included different kinds of adverbs. In both cases the 
German teachers were present in the respective schools. In the first scenario students did the task from their 
usual classroom, in the second scenario, both classes had moved to the telepresence room(s). In the second 
scenario only a smaller group of the students participated, as the teachers had chosen to divide the class into 
two groups of which only one was taught German. 

5.1 Student to Student Interaction through Telepresence 

Communicating with authentic others through telepresence can support learners’ communicative approach to 
the language and engage learners in the target language. According to Arnold. Cayley and Griffith (2002) and 
others, enhanced language learning is one of the significant impacts of videoconferencing. Thus, language 
learning and intercultural understanding is mentioned frequently in the literature as an aspect of schooling 
that videoconferencing may enhance. However, in language learning specifically, the anxiety of 
communicating with native speakers or strangers in the novel environment may intervene with the students’ 
ability to profit from the synchronous event (Kinginger 1999). In this way, the specific practices and contexts 
of language learning will affect how videoconferencing becomes part of the learning. 

In the first scenario FaceTime interaction was, as mentioned above, chosen as a framework for the task, 
as new learners of a foreign language often are reluctant to speak up in a plenary session where strangers are 
present. One of the principles underlying the use of interaction through the tablet screen was therefore to 
create a feeling of equity and proximity between the students. This was done both through the choice of 
subject - a presentation (in German) of the students’ families and near relations - and through the 
pedagogical set-up which was student-to-student interaction through the small screen of the iPad. The iPad is 
arguably a more intimate platform for videoconferencing, as the iPad is a personal device and supports a 
close up view of communication partners. 

In school T students were placed in groups of two and given the necessary contact information to call up 
the students at school A. In school A the teacher had asked the students to work individually with the task to 
avoid distractions, which meant that the interaction at the outset was a situation that was organized 
differently by the two teachers. 
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Figure 2. iPad interaction in school T and A 


Though the videoconference setting itself - as can be seen in figure 2 - is a more intimate scenario than 
the geography lesson described above, issues of remoteness and proximity still affected the interaction of 
students in the learning process. As students in the two schools did not as a rule know each other and were 
not used to collaborating, interaction never became natural or fluent, and students generally clung to their 
pre-prepared manuscripts, transforming the situation into a shared reading event rather than a conversation. 
As interaction was generally based on transmission of information, a number of the students increasingly 
became unfocused and dissociated from the task in the course of the lesson. In addition to this the 
unfamiliarity of students with this way of learning the language affected their behavior in the situation, which 
according to the teachers was to a large extent disruptive and lacked (n)etiquette. 

5.2 Teacher to Student Instruction through Telepresence 

In the second scenario teaching German grammar one-to-many became a practice of language learning that 
contributed to shaping the experience of videoconferencing as a learning event. Contrary to the practice of 
speaking a foreign language, learning German grammar was configured as a big screen one-to-many learning 
event, where students in both schools were moved to an auditorium-like classroom and situated in rows. In 
this session the teacher from school N taught the principles of adverbial connections through a PowerPoint 
presentation on the screen. The teacher at school T was present as an assistant teacher to her own class. 



Figure 3. The telepresence auditorium at school T 
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Figure 4. The remote teacher at school N 


The lesson was, I was told by the teacher, a relatively classic enactment of a grammar session, where an 
aspect of the lesson was to try to experiment with the relationship between students in school T and their 
relationship to the teachers in respectively school T and N. The question was whether students would feel 
confident with addressing the remote teacher in German in the course of the lesson. 

As remote participants in the grammar lesson, students were at the outset placed as spectators to the 
presentation given by the remote teacher. However, after her presentation students in school T were invited to 
participate in the discussion of translations of Danish sentences into German. In this context the students 
were extremely reluctant to address the remote teacher and to speak out in the plenary in German. As a rule, 
students addressed their local teacher, and spoke out to her, though she was purposely seated at the row 
behind them. In the evaluation after the session, students generally said that they had enjoyed the lesson as a 
break from classroom based teaching, and that they felt the remote teacher had presented the grammatical 
issues clearly. However, they all felt uncomfortable with speaking out in the plenary, as students in school N 
were strangers to them and they were afraid of making mistakes in the plenary. In this sense they were 
primarily placed in the telepresence event as spectators, and chose to act within their local configuration of 
teacher-student relationships. 


6. DISCUSSION AND SOME CONCLUSIONS 

The scenarios above illustrate some of the challenges, issues and potentials involved in connecting schools 
through telepresence understood as different kinds of mediated practices in everyday schooling. Whereas 
videoconferencing - according to the literature - has extensive potential for redistributing learning resources 
in rural communities and enriching learning - actually making the connection(s) is, as the examples show, 
still affected by both logistics and issues of proximity and distance. In terms of logistics, the schools in 
question must - as shown above - aim to adapt their organizational and cultural specificities to the 
community of schools with regard to for instance timetables and teaching styles. When this adaptation is 
successful, three school organizations may to some extent become one and can act as one in the synchronous 
learning environment. What works against this, however, are not only organizational structures in the 
individual schools, but also the lack of familiarity and feeling of community between students and teachers 
who are in many senses effectively strangers inhabiting the same local municipality. Therefore, connections 
between students and teachers in the schools must in future be supported and enhanced in order to establish 
engaging learning forms in the mediated environment that can create this feeling of community. 

Another issue that acts in the telepresence with iPads activities is the co-presence of several teachers and 
learners in restricted synchronous learning environments. As argued by the literature, synchronous interaction 
can be an extremely challenging framework for learning in terms of both logistics and effectively making the 
connections that support interactive learning practices (see for instance Anderson & Rourke 2005). This is an 
aspect of videoconferencing that schools should address and take seriously in their development of 
telepresence activities. In the schools in question, one of the challenges identified by the research is exactly 
these brief and fragile synchronous learning spaces that to a large extent fail to connect distributed students 
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and teachers. Therefore, the synchronous activities in these schools might be enhanced by extending and 
supporting them through asynchronous and face-to-face activities that could establish a more solid and 
meaningful basis for collaboration by bringing real connectivity back into the picture. This will entail looking 
at collaboration - and telepresence - as a broad framework for learning between the schools, in which 
different kinds of connections - both synchronous and asynchronous, mediated and face-to-face, large screen 
and desktop interaction, can support the placement of schools within the community and in the global context 
of learning. One of the exciting aspects of telepresence is thus the ways in which the potential of mediation 
changes the directions of learning - i.e. brings them out of the brick and mortar environment and into 
communities that are both global and local and in which collaboration is directed at real world communities. 
One of the interesting issues to be taken up by the research described in this paper will therefore be a further 
study of the consequences of these new directions and connections of learning, i.e. how a deeper study of 
these connections between schools and communities can bring more learning potentials into the picture. 
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ABSTRACT 

This paper presents the methodology for strategic decision making in higher education (HE). The methodology is 
structured as a cycle of strategic decision making with four phases, and it is focused on institutional and national 
perspective, i.e. on decision making that takes place at institutions of HE and relevant national authorities, in case HEIs 
are public. Case study of e-learning implementation in HE is presented. E-leaming refers to technology enhanced 
learning, blended learning, open and distance learning as well as massive online courses (MOOCs) delivered by HEIs. 
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1. INTRODUCTION 

Higher education’s (HE) key mission is to develop human potential with beneficial effects for social and 
economic development. Higher education institutions (HEIs) have been evaluated through the performance in 
triple mission: teaching, research, serving to society (outreach). Until the end of the 19 th century, universities 
were mostly focused on teaching along with conservation and dissemination of knowledge. At the end of the 
19 th century, universities directed their mission to teaching and research. Adding the so-called third mission, 
which refers to the contribution to the society, economy as well as personal development, happened at the 
end of the 20 th century. Since then it has had a remarkable influence on the development of the universities 
and HE in general. The way in which HEI meets its mission is becoming a first-class research problem 
(Cortes- Aldana et al. 2009). HE as a system, along with HEIs, must be innovative and strategically managed 
to be able to fulfil its mission. Therefore, the decision making (DM) accompanied by evaluation of its 
implementation in higher education is an issue of the highest priority. 

This paper presents the methodology for strategic decision making in HE. The methodology is focused on 
institutional and national perspective, i.e. on decision making that takes place on institutions of HE and 
relevant national authorities, in case HEIs are public. Application and adjustment of the proposed 
methodology of strategic decision making in HE is given on the example of implementation of e-learning. 

Many different terms have been used in the literature to describe learning supported by information and 
communication technology (ICT). Here, we use the term e-learning in a broader sense to cover a variety of 
forms; starting from using ICT in classrooms, blended learning, open and distant learning, online learning as 
well use of MOOCs, e-portfolios, social media technologies, open badges, etc. 

The methodology is structured as a cycle of strategic decision making with four phases: 

(1) Identification and research of the problem - problem identification and research of the problem, 
adjustment and implementation of methodologies such as readiness assessment and diffusion of innovation 

(2) Design of decision making methodology - improvement of multi-criteria decision analysis enabling 
better strategic decision making in HE 

(3) Implementation and monitoring of strategic decision - individual/group decision making supported by 
multi-criteria decision making methods, implementation of decision, identification of key factors that 
determine effectiveness of strategic decision, their correlation and design of the measuring model for 
maturity and effectiveness of strategic decision implementation in HE 
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(4) Evaluation of the strategic decision effects - evaluation by standard methods such as stakeholder 
evaluation, alignment with strategic goals but also by application of structural causal models for 
identification of strategic decision effects (Pearl, 2000). 

These four phases will be explained in the paper, and a case study of e-learning implementation will be 
presented. The case study’s complex nature, while technology is an obvious component of e-learning and 
educational pedagogy, must also focus on participation (Tucker, Gentry, 2009). There are also specifics of 
HE environment that have to be taken into account when analyzing strategic decision making. Most HEIs 
have a certain level of institutional autonomy, but also the academic freedom of researchers and professors to 
research and teach due to their best knowledge and according to their own discipline values and personal 
academic integrity. Besides that, HE can be seen as private or public good, and this introduces different 
stakeholders into the process of decision making (Marginson, 2011). However, the decision making process 
at HEIs can be very complex and essentially different from decision making in other private or public 
organizations. The public good, the historic purpose such as universities and colleges, is replaced in many 
HE systems by the demands of private interests and economy in many HE systems. However, it still has 
considerable influence on governing the HEIs. In this paper we are going to refer mostly to HEIs that have a 
certain level of autonomy from state, but are seen as a public good where staff and students participate in the 
process of strategic decision making. 


2. STATE OF THE ART 

There are different approaches to strategic planning and strategic decision making that are applied within HE. 
Lerner (Lemer, 1999) gives a review of access to the strategic planning process in HE and presents process 
model of strategic planning and decision making in HE. Related to the methods of the strategic planning and 
decision making, GAP analysis, SWOT analysis, method five competitive forces, methods of strategic 
programming and some other methods are applied. Further, Clayton (Clayton, 1997) proposed Delphi method 
as the method for making strategic decisions in education. In the paper, Jani method (Jani, 2013) based on the 
theory of inventive problem solving (TRIZ) is modified to become suitable for problem solving in HE. 

In the book (Saaty, Cillo, 2008) analytical network process (ANP) was used for decision making in the 
field of education, but mainly for individual decision making. For example: to make personal decisions about 
the choice of direction for further education and for ranking of HE institutions. Possibilities of improving 
AHP and ANP methods for multi-criteria group decision making, with applications within HE, are explored 
in the paper (Begicevic, Divjak, 2006). The methodological framework for strategic planning and decision 
making of e-learning has been developed and published in the paper (Begicevic et al., 2007). However, this 
methodology has been developed for a narrower application than the methodology presented in this paper. 

According to Okumus (2001), there are 10 key factors that affect the implementation of the strategy: its 
formulation, uncertain environment, organizational structure, organizational culture, operational planning, 
resource allocation, individuals, controls and outcomes. Key factors that influence strategic changes at 
universities are leadership, decision-making procedures, communication and evaluation (Stensakers et al., 
2014). Application of DEA method (Data Development Analysis) in the evaluation of the effectiveness of 
education is presented by (Jablonsky, 2011). We can emphasize several models for estimation of some 
aspects of e-learning effects: Anderson’s conversational e-learning model (Anderson. 2007), decision making 
model on introduction of e-learning in HE (Begicevic et al., 2007), EVEDIN model for evaluation of effects 
of learning and teaching in e-learning systems (Grubisic et al., 2007). None of these offer a complete 
framework for estimating the effects of strategic decisions in e-learning. According to Mahsood and 
Chenicheri (Mahsood, Chenicheri, 2013), literature on development and implementation of strategies in 
universities is limited. In that respect this paper contributes to the research in the field of strategic decision 
making in HE. 
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3. STRATEGIC DECISION MAKING CYCLE IN HIGH EDUCATION 

Initial pattern in developing methodology for strategic decision in HE is a methodological framework called 
Denting cycle (PDCA circle) and Dyer’s model (Dyer, 1991) of decision making and implementation. 

According to Dyer, there are four phases of decision making and implementation process: Intelligence 
phase. Design phase, Choice phase and Implementation phase. The tools that can be used in these phases are: 
data acquisition, storage and retrieval, data base management systems, interactive query, data bases, data 
analysis and decision analysis (Dyer, 1991). 

Denting cycle implies constant improvement of the system’s ability, which is the aim of the quality 
management. This cycle recognizes four phases: P (plan) - determination of the mission, vision and strategy, 
planning and establishing of objectives; D (do) - applying the processes, performing; C (check) - supervising 
and measuring of the process and their results considering objectives and indicators; A (act) - improvement 
of the process. The argument for choosing this methodological pattern is that Deming cycle is widely used in 
different areas, in its original or modified form. Besides being an effective process improvement guide it 
offers a systematic improvement method. The Deming cycle informs future improvement by providing 
feedback and maintains order during strategic planning, decision making and problem solving. It is useful for 
daily routine management, problem-solving process, project management, continuous development, vendor 
development, human resources development, new product development and process trials, etc. 

In our methodology, the Deming cycle was modified as shown in Figure 1 . The methodological cycle for 
strategic decision making in HE was upgraded with the application and adjustment of the methodology on 
case study of e-learning implementation in HE. It consists of two wider goals as two concentric cycles; the 
inner cycle representing different phases of general methodological approach described below, and the outer 
cycle representing case study of e-learning implementation in HE described in the following sections. Further 
iterations may add additional cycles representing new use-cases of given methodological framework 
(technology transfer, human resources development, inclusive curriculum development etc.). They can 
provide validity checks, as well as motivation for methodology upgrade. 



I. Development of a methodological framework for 
strategic decision-making in higher education 

m Application, adaptation and evaluation of 

methodological framework in case of e-learning 
(including open and distance learning, ODL) 
implementation 

1. Identification and research of the problem 

2. Design of decision making methodology 

3. Implementation and monitoring of strategic 
decision 

4. Evaluation of the effects of strategic decision 


Figure 1. Double cycle of strategic decision making - case study of e-learning (including ODL) 


The cycle of strategic decision making, in our case it is the one in HE, consists of four phases: (1) 
Identification and research of the problem, (2) Development of the methodology of strategic DM, (3) 
Implementation and monitoring of strategic decision and (4) Evaluation of the effects of strategic decision. 

In the first phase called Identification and research of the problem, the problem is going to be 
identified and the research of the problem, using situation analysis, conducted. In this phase the methodology 
for institution readiness assessment can be applied. Further, an additional modelling - the diffusion of 
innovation (DOI) developed by Rogers (Rogers, 2003), is valuable in this phase. According to the theory of 
DOI, diffusion is a process for transfer of innovations through certain channels, over time and among the 
members of a social system. Thereby, innovation implies every “idea, practice, or project that is perceived as 
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new by an individual or other unit of adoption” (Rogers, 2003). In this phase key factors for assessment of 
organization’s readiness for a decision about adoption of innovation and key performance indicators for the 
measurement of diffusion of innovations performance in a specific implementation case can be defined. 

In the second phase called Design of decision making methodology, based on the improvement of 
multi-criteria decision analysis enabling, better strategic decision making must be done. The methods used in 
this phase are multi-criteria decision making methods (MCDM) such as the AHP (Analytic Hierarchy 
Process) and the ANP (Analytic Network Process) (Saaty, 2001). Other methods can be also used, for 
example PROMETHEE and ELECTRE, or some of the ideal point-based multi-criteria decision methods 
(Estrella et al. 2014). In our experience and research it is convenient to use the AHP and ANP BOCR 
(Benefits, Opportunities, Costs, Risks) models for group decision making (Begicevic et al., 2007). The results 
can be analyzed by using sensitivity analysis. According to the purpose and aims of decision making, the 
multi-criteria variant of cost-benefit and risk analysis can be used. 

In the third phase called Implementation and monitoring of strategic decision, decision making 
supported by developed decision making methodology is conducted. This methodology helps decision 
makers to prioritise objectives and alternatives. The objectives and alternatives are presented in a hierarchical 
(AHP), network BOCR (ANP) structure or in a decision making table (PROMETHEE or ELECTRE). 
Decision makers are able to drill down to their level of expertise, and apply judgments to the objectives and 
alternatives considered important for achieving the goals as well as prioritize alternatives. 

In the process of group DM, the alternatives are prioritized and decision is made and implemented based 
on the obtained results. Parallel with the implementation, it is crucial to have a monitoring of the 
implementation of decision. To ensure monitoring, it is important to identify factors that determine 
effectiveness of implementation, their correlation and design of measuring the model for maturity and 
effectiveness of the decision implementation (KPI - Key Performance Indicators). In order to build a model 
for the assessment of the organization’s maturity to implement and monitor the decisions, BSC Balanced 
Scorecard. Enterprise Architecture for BPM or CMMI (Capability Maturity Model Integration) can be used. 

Monitoring the process of implementation of the strategic decision is based on milestones and other 
measurable indicators. Monitoring and evaluation are strongly linked and must be both based on reliable, 
accurate and updated data. Therefore, the purpose of monitoring is to collect relevant data to demonstrate the 
results of strategic decision implementation. The purpose of the evaluation is a rigorous and systematic 
assessment of the strategic decision implementation’s success. 

In the fourth phase called Evaluation of the effects of strategic decision, several standard qualitative 
(econometric analysis, cost-benefit analysis, multi-criteria analysis, regression analysis etc.) and quantitative 
(stakeholder perspective, document analysis, internal consistency of the strategy and external effectiveness, 
benchmarking, in-depth case study, Delphi panel, etc.) methods can be implemented, but some innovative 
approaches such as Pearl’s structural causal models for the identification of the effects of strategic decision. 
Evaluation phase also encompasses feedback that informs policy makers and is used as an input for starting a 
new cycle of strategic planning. For the evaluation to be meaningful, the strategic goals must be SMART, 
meaning that the objectives are: Specific, Measurable Achievable, Relevant and Time Bound. 

In the following sections the main stages of the outer cycle with emphasis on the e-learning 
implementation in HE are described. 

3.1 Identification and Research of the Problem - e-Learning in HE 

In general, the identification and research of the problem phase consists of three parts: (1 ) Analysis of supply 
and demand using situation analysis, (2) Determination of diffusion indicators and success, (3 ) Determination 
of readiness for innovations. 

Situation analysis includes the analysis of present state, research of the problem and analysis of potential 
solutions. The methods which can be applied at this stage are: recording of the current situation, case study 
research and different types of qualitative analysis. These methods enable identification of key factors related 
to the identified strategic problems in HE and define the responsibility for solving those problems. 

When modelling the diffusion of innovation, structural equation modelling (SEM) and social network 
analysis (SNA) can be used to identify: 1) key factors for assessment of organization’s readiness for a 
decision on the adoption of innovation in a particular social environment, and 2) key performance indicators 
for the measurement of diffusion of innovations’ performance in a specific example on e-learning 
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implementation. We emphasize here the theory of diffusion of innovations according to Rogers (Rogers, 
2003), which is almost completely ignored within the existing research. 

In this phase, the methodology for readiness assessment, for the purpose of strategic decision making, 
used to be applied. In the context of our case study of e-learning implementation in HE, the starting point is a 
methodology for e-readiness assessment based on CID methodology. CID methodology was developed by 
the Center for International Development at Harvard University (CID, 2006). It is a diagnostic instrument for 
the assessment of society’s readiness to use ICT, and it was originally based on indicators in five categories: 
access, education, society, economy and policy. Upgrade of CID methodology is necessary in order to 
accommodate the problem’s needs and to cover, at least, the following areas: Access to technology; Digital 
literacy and willingness to change of staff and students; Expectations from society and economy; Educational 
goals, policy and their funding. The co-authors of this paper conducted an extensive research on e-readiness 
and adapted the methodology in Kosovo from 2010 to 2011, and accordingly developed the strategy of 
e-learning for pre-tertiary education within the EU IT Pilot Project in the Field of Education in pre-tertiary 
education in Kosovo (Divjak, Begicevic Redep, 2012). 

Methodology for e-readiness assessment can be complemented by the use of qualitative analysis, in-depth 
interview with stakeholders, focus groups or panel discussion. This is very important for HEIs due to 
institutional autonomy and participation of staff and students in strategic DM. Therefore, raising awareness 
of new challenges and potential benefits is at the very core of strategic DM in HE. A HEI finishes the first 
phase with understanding the objectives or reasons why an organization wishes to establish e-learning 
education, and the support garners from the various institution stakeholders (Tucker, Gentry, 2009). 

3.2 Design of Decision Making Methodology - e-Learning in HE 

The development of methodology for strategic DM in HE covers three basic stages: 1 ) Choosing the most 
appropriate multi-criteria decision making methods (MCDM) related to the problem characteristics, 2) 
Improving methods for MCDM, and 3) Building models for decision making using MDCM. 

The main hypothesis at this phase is that by adjusting and implementing the methodology of multi-criteria 
decision making analysis (MCDM), it is possible to improve the quality of strategic decisions in HE. 

Decision making in HE has several perspectives and dimensions. The main perspectives are personal, 
institutional and public. Key factors that make decision making in HE more difficult are: complexity; weak 
structure and interdependence and decision making at multiple levels in the hierarchy (national, university, 
faculty/institute, department/division, personal level). The criteria by which it is possible to describe 
alternatives for making strategic decisions in HE have both qualitative and quantitative characteristics. 

Considering the characteristics we described, it is desirable that the methods for the analysis of decisions 
in HE have the following features: support for multiple criteria decision making, support for structuring of 
decision problems, support for modelling dependencies between the elements of decision making, the 
intensities of the criteria have to be expressed in qualitative and quantitative scales, support for group 
decision making, performing sensitivity analysis and risk analysis, as well as cost-benefit analysis. Existing 
methods for multi-criteria decision making only partially meet these requirements. Therefore, it was 
necessary to improve the methods for decision analysis in order to fit strategic decision making in HE. The 
starting points were methods for multi-criteria decision making: AHP and (Saaty, 2001). AHP provides a 
hierarchical structuring of decision making problems, and ANP models provide interdependencies of the 
problem via network. The AHP and ANP can be used to create a BOCR (Benefits, Opportunities, Costs, 
Risks) decision making models for group decision making on problems in HE. Other methods available are 
PROMETHEE, ELECTRE, and the ideal point-based multi-criteria decision methods and they can be used if 
they are more appropriate for solving a specific problem in HE. Due to the importance of cost-benefit 
analysis for decisions making in HE, the multi-criteria variant of cost-benefit analysis can be useful. If risks 
are identified in making strategic decisions in HE, it is important to have a risk management strategy. 
Activities in risk management can be prioritized by hybrid methodology. Monte Carlo simulation can be used 
for the analysis of financial risks. Risks associated with prioritization can be analysed by sensitivity analysis. 

In the stage of building a model for decision making using MCDM, models that incorporate methods 
from previous stage are built. In stages 2) and 3), the qualitative analysis, survey, factor analysis and 
clustering (to determine participants in the process of group decision making) can be applied. 
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3.3 Implementation and Monitoring of Strategic Decision - e-Learning in HE 

In the phase of implementing a strategic decision, individual or group decision making must be conducted 
and the made decision implemented. We propose the decision making methodology designed for strategic 
decision making in HE is based on the MCDM methods. In that context we have already mentioned AHP and 
ANP BOCR methods as the methods used for building the decision making models. However, other MCDM 
methods can be used according to the analysis in the previous phase. 

The AHP/ ANP BOCR models helps groups to structure decision into objectives and alternatives, 
prioritise objectives and alternatives, and justify decisions using sensitivity analysis. It is important to choose 
the participants in group decision making with specific knowledge; this makes them competent to assess and 
give judgments in the process of group decision making on some strategic issues. Results of group decision 
making are the objective’s relative significance and priorities of the alternatives, gained by judgment 
synthesis of participants included in decision making process. Sensitivity Analysis offers a stable solution, 
but it also enables the change of input figures as well as observing consequences on priorities of the 
alternatives. Sensitivity Analysis is used to investigate the sensitivity of the alternatives to changes in the 
priorities of the objectives. In the process of group decision making based on the developed models, the 
alternatives are prioritized and the best alternative is implemented. Parallel to the implementation, it is crucial 
to have monitoring of strategic decision. 

The implementation of strategic decisions, along with strategic goals, requires the adoption of related 
action plan which includes: activities, responsibilities, deadlines, costs, indicators and prerequisites for 
implementation. In general, the methodology is based on the following methods and models: 1) BSC 
Balanced Scorecard: in order to define indicators i.e. KPIs (Key Performance Indicators) to monitor the 
implementation of strategic decision; 2) Enterprise Architecture for BPM (Business Process Management) 
for modelling and management of activities; 3) CMMI (Capability Maturity Model Integration) to assess the 
level, of process capability and maturity of organizations for the implementation of strategic decisions. 

In order to build a model for assessment of the maturity of an organization for implementation and 
monitoring of strategic decisions in HE, CMMI model can be used. Accordingly, the relevant criteria for 
assessment of capability levels for the process of implementation and monitoring of strategic decision in HE 
must be defined. Process capability levels are the basis for determining a maturity level of the organization, 
which is an indicator of its readiness to implement the strategic decision. 

Additionally, the success of the implementation and monitoring of strategic decisions in HE can be 
assessed by using KPIs (related to BSC and BPM) and econometric methods (ROI, productivity, efficiency, 
profitability, etc.). This must be done with significant adaptation to the environment in order to interpret what 
notions of productivity, efficiency and specially profitability mean in HE. These indicators are related to 
three important project-oriented objectives: time, quality and cost. 

3.4 Evaluation of the Effects of Strategic Decision - e-Learning in HE 

The evaluation of the effects of a strategic decision in HE is very rarely researched or even performed on 
systematic basis in HEIs. However, the results of such evaluation should have a major impact on the 
following cycle of strategic planning. Therefore, the methods for strategy evaluation must be carefully 
selected and applied, and the evaluation results must be announced and used as lessons learned. In the phase 
of evaluation, different quantitative, qualitative and mixed methods can be used. The quantitative methods 
that can be used are econometric analysis, cost-benefit analysis, multi-criteria analysis and regression 
analysis. The qualitative methods that can be used for the evaluation of the effects of strategic decision are 
stakeholder perspective, document analysis, internal consistency of the strategy and external effectiveness, 
benchmarking, in-depth case study and Delphi panel. In our methodology we also suggest using some 
innovative approaches, such as Pearl’s structural causal models to identify the effects of strategic decision. 

Pearl’s structural causal models can be used for the identification of strategic decision effects. J. Pearl’s 
interpretation of causality enables quantitative modelling of causal concepts in dynamic systems. After the 
groundbreaking work in Bayesian networks and probabilistic reasoning. Pearl has been developing his 
structural causal models (SCMs) for the last 20 years. Using SCMs, we replace informal causal utterances 
with probabilistic queries. For example, we can say something about “why the system behaves in certain 
way” (explanation) or “how would the system behave under external actions” (prediction). 


24 



International Conference e-Leaming 2015 


The research on this paper topics continues in the scope of the project “Development of a methodological 
framework for strategic decision making in higher education - a case of open and distance learning 
implementation” (HigherDecision) supported by Croatian Science Foundation and planned for the period 
2015-2019. Primary goal of HigherDecison project is to develop a complete methodology for strategic DM 
and the monitoring of its implementation in HE. Two basic components of the project are: 1. Development of 
methodological framework for strategic DM and monitoring of its implementation; 2. Application, adjustment 
and evaluation of methodology on the example of decision implementation on e-learning (ODL). 


4. CONCLUSION 

ITEIs today are facing many challenges, but also vast opportunities for change. The core question is how to 
use the opportunities despite restrictions in funding, international competition and internal resistance to 
change in order to better fulfil their fundamental mission. One of the biggest opportunities is the 
implementation of educational technologies that can serve many institutional goals, such as enhancing quality 
of teaching and learning, strengthening outreach and also raising profits. However, the process of strategic 
DM is even more complex in HEIs than in other organizations due to the nature of HE system and its role in 
the development of society, economy as well as in students’ personal development. In order to systemize and 
improve the process we proposed the methodology called strategic decision making cycle that includes four 
phases. We also proposed the listed methods that can be used in each phase, as well as some specifics of 
decision making in HE and especially when e-learning is concerned. A summary is given in Table 1. 


Table 1. Summary of strategic decision making cycle in HE 


Phase of the 
cycle 

Approaches 

Specifics of HE 
and e-learning 

Methods 

Identification 
and research of 
the problem 

Needs and 
situation analysis 
Readiness 
assessment 
Diffusion of 
innovation 

Stakeholders’ 

involvement 

E-readiness 

Consciousness 

raising 

Situation analysis 

Case study research 

Different types of qualitative analysis 

Structural Equation Modelling (SEM) 

Social Network Analysis (SNA) 

Upgraded CID methodology for e-readiness assessment 

Development 

of 

methodology 
for DM 

Analysis of 
potential 
solutions 
MCDM 

Cost-benefit and 
risk analysis 

Benchmarking 
of HEIs 
Modelling 
dependencies 
and group DM 
(AHP & ANP 
with BOCR) 

BOCR AHP and ANP, PROMETHEE, ELECTRE 

Ideal point-based MCDM 

Multi-criteria variant of cost-benefit analysis 

Hybrid methodology of risk management - Monte Carlo 

simulation and Sensitivity analysis 

Different types of qualitative analysis 

Factor analysis, Clustering 

Implementation 
and strategic 
decision 
monitoring 

BSC, KPI, BPM 

CMMI 

PPM 

Interpretations 
of econometrics 
and use of KPIs 
and PPM 

BSC Balanced Scorecard 

Enterprise Architecture for BPM (Business Process 
Management) 

CMMI (Capability Maturity Model Integration) 
Econometric methods (ROI, productivity, efficiency, 
profitability) 

Evaluation of 
effects of the 
strategic 
decisions 

Qualitative, 
quantitative and 
mixed methods 
Structural causal 
models 

Stakeholder 
perspective 
analysis 
In-depth case 
study to find out 
causes & effects 

Qualitative methods - stakeholder perspective, document 
analysis, internal consistency of the strategy and external 
effectiveness, benchmarking, in-depth case study, Delphi 
Quantitative methods - econometric analysis, cost-benefit 
analysis, multi -criteria analysis and regression analysis 
Innovative approaches - Pearl’s structural causal models 


Finally, many open questions still remain and therefore the need for further research. In the phase of 
problem definition and exploration different stakeholders must be involved in order to contribute to 
development of goals and alternative solution. Different stakeholders have diversified goals and expectation 
of HE. However, if group decision making is facilitated in effective and efficient manner, solution can be 
reached and implemented. Nevertheless, implementation monitoring and evaluation are neither a common 
practice in HEI nor a well-researched area. Finally, the proposed research could have a reach beyond HE. 
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ABSTRACT 

Adaptive eLearning systems are able to adjust to a user’s learning needs, usually by user modeling or tracking progress. 
Such learner-adaptive behavior has rapidly become a hot topic for eLearning, furthered in part by the recent rapid 
increase in the use of MOOCs (Massive Open Online Courses). A lack of general, individual, and situational data about 
student populations currently hampers the infusion of effective adaptive behavior into existing eLearning platforms. This 
contribution presents original research on using differences in individual learning styles. Factors related to performance, 
motivation, satisfaction, and previous knowledge were targeted and used to assess the effectiveness of the approach. We 
discuss alternative bases for adaptation (e.g. cognitive styles), style distributions in student populations, and conclude 
with repercussions for adaptive behavior in HCI in general. 


KEYWORDS 

Adaptive eLearning, Learning Styles, eLearning, Inter-individual Differences, Learner Satisfaction, Motivation 


1. INTRODUCTION 

Recently, MOOCs have seen a rapid increase in use, both in the number of courses that are on offer, often for 
free, as well as in attendance per course, with participant numbers surpassing 230,000 for individual courses 
(Jordan 2014). Additionally, students are drawn from increasingly heterogeneous population groups (e.g., 
regarding age or cultural and educational backgrounds) (Ha 2014). There is every indication that such 
development will be continuing for the coming years, as additional universities, companies, educational 
alliances, and funding agencies are increasing their commitments, and as MOOCs (along with eLearning 
systems in general) are frequently seen as a suitable instrument to further social inclusion on national and 
international levels. Last, the continuing increase in the use of mobile devices can be expected to additionally 
increase the use of eLearning, as learning can spread to more diverse settings, contexts, and locations outside 
the traditional classroom. This creates new challenges for human -computer interaction in eLearning systems. 

Compared to traditional, classroom-based learning, eLearning is characterized, among other differences, 
by a dramatically decreased instructor/student ratio (with less time and less other resources available per 
student) and a dramatically increased variation in the student population and in contexts and patterns of use. 
In brief, there exists less option for the instructor to adapt and mediate a course’s content to the individual 
student, and more need to do so. This is a gap that eLearning systems need to increasingly fill. In the fields of 
CSCW and Ubiquitous Computing, there has been a significant amount of research on systems that leverage 
communication or exchange between learners and instructors, or among learners, aiming to bridge this gap 
(e.g. Abowd et al. 1999, Di Cerbo et al. 2010). In such systems, chats, message boards, or shared documents 


1 Excerpts of this paper have already been presented as a short article (poster) at the Human Computer Interaction Conference, Munich, 

in September 2014. 
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help to create social awareness and ease getting in touch with fellow learners. Other contributions pointed to 
beneficial effects of guidance or framing in facilitating collaboration among learners in MOOCs 
(Kizilcec 2013). While such approaches certainly represent important contributions to effective eLearning in 
general, we believe that a promising approach to filling the gap lies in tailoring or mediating the standard 
content of online courses such that it will adequately fit and adapt to students’ individual backgrounds and 
learning styles. The goal is to maintain efficiency and effectiveness of the individual learning experience. 

Adaptivity seems of specific importance for eLearning, due to learners’ differences in goals, learning 
styles, previous knowledge, and backgrounds, and because eLearning systems permit creating individual 
learning paths (Brusilovsky 1996). Adaptive eLearning systems should be able to respond to individual 
variables and provide a personal access to learning material (e.g. by adapting it to individual needs). This 
contrasts with the current state of eLearning systems, in which material is largely available in the same 
format for all learners (Hauger & Kock 2007). 

A number of important learner variables to which a system can adapt address cognitive factors. Mayer 
and Massa (2003) differentiate between cognitive abilities (i.e. what people are capable of doing), cognitive 
styles (i.e. ways in which people process and represent information), and learning preferences (i.e. ways in 
which people like information to be presented to them). We will employ this categorization, while referring 
to Mayer and Massa’s learning preferences as learning styles. One should take note that the terms cognitive 
and learning styles are not used consistently in the literature, nor are they always precisely defined (Riding & 
Cheema 1991). Riding and Cheema describe learning styles as a type of cognitive style for a learning 
situation. Hartley (1998) considers learning styles as the learner’s individual way to solving a learning 
exercise. The terms are used in educational psychology to describe inter-individual differences in learning 
contexts (Dung & Florea 2012). Such inter- in dividual differences in abilities and styles correspond to the tier 
of individual factors described by Bertel (2010) in a tiered factor user model for joint human-computer 
reasoning tasks. 

This paper presents original research on whether an adaptation of eLearning material to inter-individual 
differences in learning styles can increase learning effectiveness and efficiency, learner motivation, and 
learner satisfaction. We will present a behavioral study and analyze and discuss resulting data, also with 
regard to alternative bases for adaptation, such as cognitive styles, and to properties of distributions of styles 
and preferences in relevant student populations. The paper will conclude with a discussion of possible 
repercussions for adaptive behavior in human -computer interaction in general and of future work. 


2. METHODOLOGY AND METHOD 

For modeling learning styles on an individual basis, we chose to employ the Felder -Silverman Learning Style 
Model ( FSLSM ; Felder & Silverman 1988), which, though originally introduced as a model for engineering 
classes, has been suggested to be well suited for use with eLearning (Carver et al. 1999; Kuljis & Liu 2005). 
Internal reliability of the FSLSM has been validated and tested in various studies (Felder & Spurlin 2005), 
and the model is based on well-established learning styles, including the Myers-Briggs Type Indicator 
(Briggs Myers & Briggs 1962) and the Learning Style Inventory by Kolb (1984) (Felder & Silverman 1988). 
The FSLSM’ s four dimensions of altogether eight paired styles are sequential/ global, sensing/intuitive, 
active/reflective and visual/verbal. To identify an individual’s learning styles, the self-reported Index of 
Learning Styles ( ILS ) questionnaire can be used (Felder & Soloman n. d.). It contains eleven questions for 
each dimension of the model. For the eight learning styles, it each distinguishes between balanced, moderate, 
and strong style expressions (cf. Figure 1). An individual learner’s expressions on each of these styles are not 
to be seen as preferences fixed for life, but as variable ones, depending on the learning context. 


verbal 


11 9 

7 5 

1 1 


3 1 

1 1 



1 3 

1 1 


5 7 

1 1 

9 11 

1 1 


visual 

1 1 

1 1 


1 1 



1 1 


1 1 

1 1 



strong, moderate, balanced learning style 


Figure 1. Visual/verbal dimension of the FSLSM and their possible values (based on Felder & Soloman n. d.). 


28 


International Conference e-Leaming 2015 


The visual/verbal dimension of the FSLSM addresses how effectively external information can be 
processed through which sensory and cognitive modal channels. A predominantly visual learner remembers 
best what she sees as pictures, diagrams, films, etc. or when she creates a visual mental image based on 
external information. A predominantly verbal learner prefers words in written or spoken form to learn. For 
instance, on the visual/verbal dimension, balanced learners are able to cope with both representation formats, 
whereas strong visual learners will likely have problems to learn with verbal representations, and vice-versa. 

Within the study reported here, we chose to focus on adapting eLearning material along the visual/verbal 
style pair of the FSLSM, in part because the dimension well -investigated, also in cognitive style research, and 
because a necessary adaptation of the presentation format of our learning material (in order to match with 
either a visual or verbal learning style, e.g. as pictures or text) could be achieved with limited resources. It 
has been argued elsewhere that ILS results can be used for creating user models in eLearning systems 
(Paredes & Rodriguez 2002), and the instrument has in fact been employed for several studies of adaptive 
learning environments (Dung & Florea 2012). Parvez and Blank (2007) argue that the FSLSM dimensions 
and their characteristics are simple enough so that an integration of ILS results into eLearning systems 
becomes feasible. Different learning objects, such as text, images, summaries, exercises etc., can be 
developed for each dimension and style of the FSLSM. 

The present study employed our own translation of the ILS into German. Internal consistency of the 
translated version was good, as Cronbach’s a for the translated version and for items of the visual/verbal 
scale was .66. This fits coefficients reported by other studies (e.g. Litzinger et al. 2007); it indicates sufficient 
reliability of the translated instrument (>0.5 for attitude test, Tuckman and Harper 2012). Additionally, we 
measured students’ individual cognitive styles via a translated version of the Revised WQ by Kirby et al. 
(1988). This is an adapted version of Richardson’s original Verbaliser-Visualiser Questionnaire (WQ, 
Richardson 1977), which, according to Jonassen and Grabowski (1993), is the most frequently used tool to 
measure visual or verbal ways of thinking. Reliability coefficients for this measure for the verbal/visual 
scales in our study were a=.59 and a=.58, respectively. Though we could trace no other measurements of 
reliability for the Revised WQ to compare our scores (other than those provided by Kirby et al.) many 
previous studies have employed the instrument (e.g. Choi & Sardar 2011). 

We deem learner’s motivation to be essential for learning outcome and believe high motivation levels to 
be an important precondition for sustained engagement in learning. Learning motivation can be labeled as 
intrinsic or extrinsic. Pure interests and curiosity have been described to persuade students to learn in a 
focused manner, requiring little external interventions for sustained learning, as learning is done mostly for 
its own sake (Brandstatter et al. 2012). In this study, we determined participants’ intrinsic motivation based 
on a self-reported questionnaire by Isen and Reeve (2005). To determine satisfaction with the learning 
material, we designed a self-reported questionnaire. 

Few previous studies have employed the visual/verbal learning style of the FSLSM in the context of 
adaptive eLearning (e.g. Brown et al. 2006). While these usually focused solely on measuring 
performance-related outcomes, we chose to take matters a step further for this study and also focused on 
emotional factors related to learner satisfaction and motivation. One can argue that, as learners are more on 
their own during eLearning than during traditional learning formats, there exists more need for sustained high 
learner motivation. Self-regulated learning is often a challenge for many learners. Intrinsic motivation and 
high satisfaction levels with the material are thus especially important. Our main research questions were as 
follows: (Ql) Is there a positive influence of a good fit between the representation format of the learning 
material and the learning style on learning performance? (i.e., do learning times decrease for constant 
performance levels? Does performance increase with constant learning times?) (Q2) Does a good fit increase 
learner motivation and satisfaction? 

To answer these questions, we designed a study based on the Moodle eLearning platform. 53 participants 
(26 female, 27 male), with a mean age of 25.3 years [20-34 years], participated voluntarily. There was no 
reward in terms of money or credits. The study was conducted under laboratory conditions at 
Bauhaus-Universitat Weimar to control and minimize effects of distraction and disruption on the study and 
on learners’ concentration. Participants were either students, mostly of Computer Science and Medici degree 
programs, or university staff. They learned at computers and were told that they would be given a quiz about 
the learning material. Participants were also provided with a questionnaire on their learning motivation and 
satisfaction. The learning material was taken from an established eLearning course intended for future civil 
engineers on the theory of oscillations, specifically from a section on the “Basics of Sound”. The chosen 
chapters were comprehensible with secondary school knowledge, and no further instruction in physics or 
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math subjects was required. The material was reproduced in two versions for the learning styles of the 
visual/verbal dimension: one centered mostly on using illustrative diagrams, the other on using textual 
descriptions. The lecturer of the original course acted as an expert evaluator and ensured that both versions 
contained the same information and that expected learning times were each at 20 minutes. 

The study was of mixed design. As the between -subject component, two groups (A & B) were formed, 
which received the same learning units, however in different display formats ( visual or verbal). We used 
Moodle to assign a participant to a group based on her individual learning style, as established via the ILS 
instrument. One half of all units were presented according to the individual learning style, the other half were 
presented in the opposite format. The within-subject component contained the questions of the quiz, and all 
participants were given the same set of questions. Based on the research questions, the independent variable 
was the display format (or, more precisely, whether the format matched an individual learner’s style, or 
whether it did not). As dependent variables, study time, test performance as well as learner motivation, and 
satisfaction with the learning material were used. In addition, participants’ individual cognitive styles and 
previous relevant knowledge were established. 


3. RESULTS 


As it turns out, three quarters of our participants had a visual learning style (in a simple dichotomy of visual 
and verbal styles). This corresponds with findings of other studies that about 74% of students ( N>2800 ) have 
visual learning styles (for an overview, see Felder & Spurlin 2005). In natural science study programs, a 
majority of students are visual (either moderate or strong), whereas the combined percentage of balanced and 
verbal learners is comparably higher in social science programs (again, see Felder & Spurlin 2005). 
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Figure 2. Distribution of the subjects on the learning styles. 

A correlation analysis and a set of ANCOVAs showed that previous knowledge had no influence on 
learning outcomes, nor did it influence motivation, satisfaction, or learning times. Data from one participant 
was removed before the statistical analysis was conducted because the boxplot of the study time identified it 
as an extreme value (with more than the triple interquartile range). Another participant did not submit the 
second part of the quiz; therefore his or her data was not included in the analysis of learning outcomes. As 
most data was found to be non -normally distributed, we used nonparametric statistical tests (such as 
Mann-Whitney-U). The two key findings of the study are that (1) there was no significant influence of a good 
or bad fit between material format and individual learning style on study time and learning outcomes, but that 
(2) there was a (often, highly) significant influence of a good or bad fit on learner satisfaction and motivation. 

The main focus of the further analysis lies on the group of moderate to strong visual learners, as they 
constituted the largest learning style group of our participants (see Figure 2). We will start by examining 
performance-related measures (i.e., study times & learning outcomes as measured by performance in the 
quiz). There were N=13 visual style learners in study group A (lesson 1: pictures, lesson 2: text) and N=10 
visual style learners in group B (lesson 1: text, lesson 2: pictures). We found no significant differences in 
study times for lesson 1 between groups A (M=13.63 min ) and B (M=10.52 min ) ( U=40.00 , Z= 1,550. ns, 
r=—.32 ), nor for lesson 2 (A: M=10.10 min, B: M=9.23 min, U=56.00, Z=— 0.558, ns, r=-.12 ). Moreover, 
we found no significant correlation between the individual cognitive style and study times in 
matched/mismatched lessons among all subjects ( N=52 ). Similarly, a comparison of learning outcomes 
between groups A and B showed no significant differences for visual learners (lesson 1: A: M=79.86%, B: 
M=75.00%, U=43.50, Z=~l,340, ns, r=-.28\ lesson 2: A: M=71.87%, B: M=72.29%, U=64.50, Z=-.031, 
ns, r=—.01). There was no significant difference in performance results between lessons that were presented 
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in pictorial and text form. There was also no significant correlation between matched/mismatched individual 
cognitive styles and learning outcomes for all participants ( N=51 ). This confirms findings by Kolloffel 
(2012) / Brown et al. (2006) who also did not find significant interrelationships between cognitive styles / 
learning styles and learning outcomes in matched/mismatched courses. 

To sum up: There were no significant effects of style match/mismatch on any of the performance -related 
measures. As we will see in the following, this does, however, not imply that style-to-content 
matches/mismatches are of no consequence for eLearning and that an adaptation of eLearning content to a 
learner’s individual style would, therefore, be unimportant. Quite the contrary: A comparison of intrinsic 
motivation levels between groups A and B after lesson 1 (again, for visual learning style participants) showed 
a significant effect of matching learning style to content format (A, pictures/match: M=12.54, B, 
text/mismatch: M=8.70, U=26.50, Z=— 2.409, p<.05, r=—.50), though no such significant effect could be 
found for lesson 2 (A, text/mismatch: M=12.23, B, pictures/match: M=10.60, U=51.00, Z=~0,873, ns, 
r=—.18). Visual participants who received pictorial material in lesson 1 were thus significantly more 
motivated than visual participants who started with the text material. Interestingly, such high motivation 
levels did not decline after the second lesson, which was presented as a mismatch (i.e., in text form). In group 
B, participants started with text; for the visual participants, motivation increased significantly after lesson 2 
in which material was presented as pictures. We believe that such dynamics of motivation level due to 
alternating matches/mismatches merit further research for adaptive eLearning in particular, but also for 
adaptive human-computer interfaces in general. One interpretation of our data is that high motivation levels 
caused by style-matched material declines slowly during a mismatch, and that it rises more quickly when 
mismatched material is followed by matched material. 

Data on satisfaction with the learning material was gathered after each lesson, together with the data 
about learner motivation that we discussed in the preceding paragraph. Again for visual participants, a 
comparison of satisfaction levels in lesson 1 showed a highly significant difference between group A 
(pictures/match, M=9.23 ) and B (text/mismatch, M=6.0 ) ( U=23.00 , Z=~2.654, p<.01, r=—.55). This group 
comparison remained highly significantly different after lesson 2, although with a further increased effect 
size (A: M=3.54, B: M=10.00, U=2.00, Z=~4.002, p<.001, r=-.83). Even within the groups, we found 
highly significant effects of a style-matched presentation format on the satisfaction with the material. A 
Wilcoxon signed-rank test showed highly significant satisfaction differences inside groups A (Z=— 3.066, 
p<.001, r=—.85) and B (Z=— 2.712, p<.01, r=—.86). Visual learners were thus highly significantly more 
pleased with pictorial material. They did not like to learn with text only. Interestingly, none of these tests 
remain significant when aggregating data from all visual learners (including balanced visual learners). This 
shows that the effects reported here are really related to an expressed visual learning style, and that learners’ 
individual strengths of style expressions may be just as important for designing effective adaptive behavior in 
eLearning systems as a sorting into dichotomous visual/verbal categories. 

A secondary finding of our study is that ILS and Revised WQ scores (for learning and cognitive styles, 
respectively) were highly significantly correlated. To compare the two models, we performed pairwise 
correlation analyses of the visual/verbal answers of the ILS with the corresponding visual/verbal scales of the 
Revised WQ (visual-to-visual: r=.84, p<.01, for all our subjects, N=53; verbal-to-verbal: r=.50, p<.01). To 
our knowledge, a significant correlation between these instruments has not been reported before. 


4. CONCLUSION, DISCUSSION AND FUTURE WORK 

The aim of this contribution was to see how adaptive behavior can be infused into an existing eLearning 
platform by categorizing learners into subpopulations according to their individual learning styles, and by 
then presenting learning material in different matched and mismatched versions for each subpopulation. To 
this end. we created variants of learning material for an established eLearning course and conducted a 
behavioral study to investigate effects of matching visual/verbal learning styles to corresponding material 
formats onto learning performance, study times, intrinsic learner motivation, and satisfaction. For the 
identification of the individual learning style, the Felder Silverman Learning Style Model was used, reduced 
to the visual/verbal dimension of the model. 
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We found no significant influences of a style-matched presentation of learning material on study time and 
learning outcome. This finding is in line with former research on this topic. However, we equally focused on 
emotional factors, such as learner motivation and satisfaction. For scores of these factors, a presentation of 
learning material that is well matched to the individual’s style of learning turned out to be highly important. 

Emotional factors should not be underestimated for eLearning, as they play a substantial role for 
self-regulated learning. As Schiefele and Schreyer (1994) demonstrated, intrinsic learning motivation is 
significantly positively correlated with measures of learning success, such as grades and test results, and high 
intrinsic motivation encourages in-depth and conceptual forms of learning. In addition. Levy (2007) 
identified learner satisfaction as a major factor in students’ decisions to complete or drop eLearning courses 
(this is, in fact, a key problem of MOOCs, specifically Jordan 2014). Even though the study on which we 
reported here did not show a direct influence of style-matched material onto learning success, one could 
argue that via the discovered strong influence of style-matched learning material on intrinsic motivation, as 
well as via the influence of intrinsic motivation on learning success widely established elsewhere, an indirect 
effect of learning style on learning success likely exists. This point clearly requires further and more 
systematic research. 

Further work is also needed to investigate the long-term impacts of style-matched courses on the 
performance-related factors, either directly or indirectly via emotional factors. We thus suggest to conduct a 
long-term study and to employ a suitable test-retest procedure that would have to be developed. Moreover, 
there is a need to conduct such study with a larger sample to verify the results obtained here also for the 
verbal learners. Ideally such investigation could be conducted in a real learning setting, as extrinsic 
motivation, such as grades, also play an important, presumptively a negative, role (Lepper et al. 2005). 

We argue that some of the findings presented here clearly have important implications for the design of 
adaptive eLearning courses, as individuals would seem to benefit from style-matched course material. A 
major factor of relevance of this contribution lies in establishing that a matched visual/verbal learning style is 
a highly significant factor for the motivation and the satisfaction of the learner. We would also argue that 
difficulties encountered in the present study by highly unbalanced distributions of learning and cognitive 
styles in relevant user population of eLearning systems pose challenges not only for developing adaptive 
eLearning systems, but for developing adaptive human-computer systems in general. When a user is 
interacting with an adaptive system on a 1-to-l basis, it is naturally inconsequential for effective adaptive 
behavior how frequently that user’s specific cognitive or other styles (or preferences) are encountered within 
the relevant user population. What remains important is that the adaptation occurs with respect to specifically 
those styles (or preferences). For investigating how adaptation should best occur based on a given user type, 
it is, however, far from being inconsequential how frequently such type is encountered. In the present study, 
we were largely unable to draw conclusions about users with verbal learning styles, as such users were too 
rarely encountered within the population of, mostly, STEM 2 students from which we sampled participants. 
Excluding verbal style learners from using adaptive eLearning systems as a result does not strike us as an 
attractive conclusion. This problem exists for interaction design in eLearning, as well as for all 
human-computer interaction types for which inter-individually differing, individual factors of a user will play 
a role. 

We believe that a viable course of action may lie in combining quantitative measures about frequently 
encountered user types with qualitative measures applied to the more infrequent types. This would have to go 
hand-in-hand with a graduated approach towards confidence that the adapting system places in its actions, as 
well with different degrees (i.e., strengths) of adaptation. A second, promising course of action could lie in 
establishing a tiered user model (along the lines sketched by Bertel 2010), in which the generation of 
adaptive behavior would resort to being based on general cognitive factors as long as more specific 
information about a current user and his or her cognitive types or preferences is unavailable. 
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ABSTRACT 

Higher Vocational Education is one of the most important educational forms. But in the course of implementation of 
Higher Vocational Education, we find three inevitable questions: Higher Vocational Educational system's length of 
schooling which is only three years is too short; the differences in personality of teachers affect the teaching effects; the 
lower quality of the students disengaged their passion to learn. MOOC is a new teaching mode which has successful 
cases such as a university computer course: a Computational Thinking Perspective of Shandong University. We refer to 
the university MOOC - a Computational Thinking Perspective, to construct a professional SPOC teaching mode and 
recur to the University MOOC in the process of teaching. Seen from the evaluation and suggests of teachers and students 
joined the MOOC and SPOC, the teaching mode not only can stimulate the teachers' teaching enthusiasm, but also can 
stimulate the students' learning enthusiasm. 


KEYWORDS 

University MOOC; Professional SPEC; Computational Thinking Perspective. 


1. INTRODUCTION 

With the fast-speed development of the modem information society, the recognition to the Higher Vocational 
Education is increasingly high. And it has reached a situation that Higher Vocational Education even is 
comparable to the University Education. Such a significant form of education also shows some inevitable 
questions when it has been in the course of implementation: 

The first question is Higher Vocational Educational system’s length of schooling. We must complete all 
the courses including theory, practice, internship, and graduation design in three years which is too short. 
And we must take into account the students' employment problems but also take into account the part of the 
students into higher education. 

The second question is the differences in personality of teachers. Most teachers in higher vocational 
college are university graduates who have no much practicing or teaching experience. And less number of 
teaches, larger number of courses, faster updating of the course contents have put forward a huge challenge 
to our higher vocational teachers. 

The third question is the quality of the students. The Higher Vocational College students have some 
generalities: lower grades, poorer foundation, and less learning enthusiasm, which cause teaching and 
learning more difficult. We must adopt better teaching methods to stimulate their learning enthusiasm, and 
then they will get more useful knowledge. 

Fortunately, a brand-new teaching mode called MOOC [1] (Massive Open Online Courses) provides a 
feasible method to solve these problems above. In this article, we construct an appropriate professional SPOC 
(Small Private Online Courses) in Higher Vocational Colleges after analyzing a university MOOC and recur 
to the university MOOC in the process of teaching of the Higher Vocational College. 


35 



ISBN: 978-989-8533-40-1 ©2015 


2. RELATED RESEARCHES 

MOOC originated in the openness of the internet based courses which can be found earlier in 2007. And in 
the next year, Pr. Alec Couros in University of Regina developed an open online course called “Media and 
Open Education (EC&I 831)”. Later, a true meaning MOOC called “Connectivism and connectivity 
knowledge” attracted 25 students and 2300 free online participation of students [2,3]. In 2012 called the year 
of MOOC, three world MOOC platforms - Coursera [4], Edx [5], and Udacity [6] began to provide courses to 
worldwide learners [7]. The NMC Horizon Report of 2013 Higher Education Edition pointed out that 
massively open online courses have received their fair share in 2012, and are expected to grown in number 
and influence within the next year [8]. Today, we have seen that the prediction has come true. 

MOOC is composed of five main elements: teachers, students, subject, learning materials and learning 
situations [9, 10]. Through the reasonable combination and application of the five elements, it has obtained 
the purpose of flipped classroom and improved efficiency, standard [11] by using MOOC. Nowadays, more 
and more courses are joining MOOCs and more and more learners are using MOOCs. 

In china, 4000 high quality courses had been constructed between 2003 and 2010. After 2011, 
accompanied by the ministry of education of the implementation of the national fine -designed open courses, 
more than 100 resource sharing lessons had been opened to public. On May 8, 2014, the Chinese University 
MOOC was online which marked the arrival of the era of a new course. 


3. A MOOC CALLED “UNIVERSITY COMPUTER FOUNDATION: A 
COMPUTATIONAL THINKING PERSPECTIVE” 


3.1 The Teaching Goals 

In the course of “University Computer Foundation”, Pro. Hao provides three teaching goals which are: 
Cultivating the students' information literacy, cultivating the students' computational thinking, and teaching 
computing science knowledge. 

3.1.1 Cultivating the Students' Information Literacy (The First Ability) 

Information literacy is the application and learning information technology in the work and finally obtaining 
the ability to solve problems by using information technology have learned [13]. The ability also tells people 
when you need information and how to find, evaluate and effectively use it. 

3.1.2 Cultivating the Students' Computational Thinking (The Second Ability) 

Computational thinking is the ability using the basic concepts of computer science technology to solve 
problems, design systems, and understand human behaviors, which covering a series of thinking activity of 
the breadth of computer science[13]. The ability also tells people how to solve problem by using systematic 
thinking. 

3.1.3 Teaching Computing Science Knowledge (The Third Ability) 

In daily life, the computer has become an indispensable facility. Mastering relevant computing science 
knowledge which including computer basic knowledge, data structure, algorithm design, and program design 
language and so on, has become the need of the times. The ability can make us well into the society, and deal 
with the problems encountered. 

Following these three progressive targets to cultivate the learners' ability, then they will well adapt to 
social work and learn etc. 

3.2 The Design of the Course 

The design of the course determines the life of the course. 
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3.2.1 Problem Exiting in Current Course 

“University Computer Foundation” is a required course to non-computer major in college and has the 
characteristics of large quantity and extent. Although has experienced several reforms, the teaching mode 
based on computer operation skill is still a serious challenge in the course offered. Around 2010, 
computational thinking for the reform of computer basis teaching had been widely recognized in domestic 
and foreign universities. And the teaching design also is welcomed by the students and the colleges. 

3.2.2 The MOOC Course Structure Design 

The course structure design facing to the real work environment comes from the following train of thought: 

* Information society needs information literacy, 

* Information literacy comes from computational thinking, 

* Computational thinking comes from computer knowledge. 

Figure 1 shows the three layer model of demand of the course structure design. 



Figure 1 . The three layer model of demand of the course 


3.2.3 The MOOC Knowledge Module Design 

Closely around the three layer model of demand, we put forward the following basic process of calculating 
by computer. 



Figure 2. The basic process of calculating by computer 

Referring to Figure 2, solving the problem of the conceptual model as the main line of computer, the 
course includes seven knowledge modules: 

(1) Introduction, 

(2) Compute and Computer, 

(3) Problem Solving and Algorithm, 

(4) Data and Data Structure, 

(5) Computer Program, 

(6) Computer Network, 

(7) Computational Science Frontier. 

In the process of knowledge module design, a number of knowledge points [14, 15] and knowledge unites 
[14, 15] are extracted to support the modules and courses. 
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4. A PROFESSIONAL SPOC CALLED “HIGHER VOCATIONAL 
COMPUTER PROFESSIONAL DESIGN: A COMPUTATIONAL 
THINKING PERSPECTIVE” 


4.1 Educational System 

The length of schooling of Higher Vocational Colleges is not enough long, why? In a Higher Vocational 
Colleges we have only four semesters to finish all professional courses and practice courses teaching, then 
the rest two semesters are internship, and graduation design. In the process of teaching, we must take into 
account the students' employment problems but also take into account the part of the students into higher 
education. The best way to satisfy this requirement is fusing a few door edge professional courses to meet the 
length of schooling. The MOOC in section 3 can finish the task. So we draw support from the MOOCs 
thought. 

4.2 The Differences in Personality of Teachers 

As everyone knows, teachers in University have doctor's degree that have strong ability of research and 
exploration including teaching and science technology, and the main goal of university is to culture talents 
with high level and knowledge. While teachers in Higher Vocational Colleges have master's degree that have 
lesser ability of research and exploration and lesser practical experience, and the main goal of Higher 
Vocational Colleges is to train skilled talents. So in Higher Vocational Colleges, we pay much attention to 
enhance teachers' practical ability and the same time we have less time to do scientific researches. So we can 
draw support from MOOC's thought which launched by university professors. 

4.3 The Quality of the Students 

Lower grades, poorer foundation, and less learning enthusiasm are the main characters of the Higher 
Vocational Colleges students. How to stimulate their learning enthusiasm is one of the most important 
problems. And how to explain the learning content is also important. The organization of the courses content 
and the proper teaching methods play an important role in the courses of teaching. Also we can draw support 
from MOOC's thought which adopted flipped classroom etc. 

4.4 A Professional SPOC referring to a University MOOC 

After analyzing the problem exiting in Higher Vocational Colleges, we propose a professional SPOC 
according to the university MOOC's idea. 

4.4.1 The Professional SPOC Course Structure Design 

According to the MOOC in section 3, the computer professional courses' structure design also facing to the 
real work environment comes from the following train of thought: 

• Information society need information literacy and information skills, 

• Information literacy and information skills come from computational thinking(software and 
hardware), 

• Computational thinking comes from computer knowledge (software and hardware). 

Figure 3 shows the three layer model of demand of the professional SPOC course structure design. 
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Figure 3. The three layer model of demand of the specialty 


4.4.2 The Professional SPOC Knowledge Module Design 

Closely around the three layer model of demand of the specialty, we put forward the following basic process 
of calculating by computer. 


Referring to Figure 4, solving the problem of the conceptual model as the main line of computer, the 
professional courses include two kinds of knowledge modules, and every module has several main courses: 

(1) Hardware Module, 

(a) Basic Knowledge of Computer, 

(b) Computer Communication and Network, 

(c) Introduction to Internet of Things, 

(d) Internet Comprehensive Wiring Engineering, 

(e) Audio and video system construction, 

(f) Computer Room Construction. 

(2) Software Module, 

(a) Problem Solving and Algorithm, 

(b) Data and Data Structure, 

(c) Database Basics, 

(d) Computer Program. 

In the process of knowledge module design, a number of relevant courses are opened to support the 
specialty. And we also join some edge courses in our main module to broaden the students' vision. For 
example, we give lectures either by ourselves or inviting scholars of famous university. 



Figure 4. The basic process of calculating by computer of the computer specialty 
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5. WHICH IS MORE SUITABLE TO OUR STUDENTS? 

We have implemented according to the design in section 3 and 4 a month long. And the course is C Language 
Program Design. We adopt the teaching method of flipped classroom as follows: 

• Students learn content by micro class video, 

• Students ask questions, teachers answer question, and students discuss, 

• Teachers interpret the micro content again, 

• Quizzing students in class and give an assignment next class. 

In a month, we have gone on Select Statement and Loop Statement of C Language Program Design and 
we have a statistic data of the course's learn state in table 1 . 


Table 1 . The learn state before and after adopting the MOOC teaching method 



Attendance! %Y 

Enthusiasm! %) b 

Quizzing result(Seore) c 

Before 

82.3 

23.0 

73.4 

After 

87.6 

43.5 

86.0 


Note'. Table notes. 

“Sample footnote A: the average of the attendance, 

b Sample footnote B: the average of the number of hands to answer questions, 
“Sample footnote C: the average of every quizzing result. 


According to Table 1, the rate of Attendance, Enthusiasm, and Score (Quizzing result) have raised much. 
We see the effectiveness of the reform jauntily. Both teachers and students benefit from the teaching method. 

There are three teaching modes which are the traditional teaching mode and the MOOC and the 
professional SPOC teaching mode. Which is the most suitable teaching mode to our students? Yes, the 
MOOC or professional SPOC teaching mode is. 


6. STUDENTS’ EVALUATION AND SUGGESTIONS 

We cannot evaluate a kind of teaching method only from the perspective of the appearance. We should listen 
to the views of students. We sent a questionnaire for teaching to make students’ feedback pointing at this 
month's teaching. We put in order students' feedback as follows in table 2. 

Table 2. Students' feedback to the MOOC and professional SPOC teaching mode 


Feedback Content 

We are willing to answer questions. 

Positive We are willing to discuss problems with teachers. 

Feedback We would like to explain the class knowledge. 

We like this lesson. 

We must do a lot of work after class. 

Negative 

& We must check the information in the library or surfing the internet 
Feedback 

We must think over more problems. 

Note: Table notes. 

The “Positive” and “Negative” are said to student.. 


In fact, the MOOCs thought which has been applied to teaching and specialty constriction is successful 
from the results of students’ feedback. All feedback is motivating their learning interests which are the 
ultimate aim of our teaching. 
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7. TEACHERS' EVALUATION AND SUGGESTIONS 

Most teachers approve the organization of the specialty and the teaching method. They can mobilize the 
enthusiasm of the students and bring passions to their own teaching process. 
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ABSTRACT 

In the context of education and training, competency measurement (CM) is a central challenge in competency 
management. For complex CMs, a compromise must be addressed between the time available and the number of 
dimensions to be measured or the quality of the measurements. Increasing the efficiency of existing tests for CMs 
therefore poses a key challenge. An important approach to this challenge is computerized adaptive testing. For CMs, 
there is currently a lack of integrated adaptive testing tools. This paper presents the implementation, integration and 
evaluation of an appropriate adaptive component for the example of the learning management system ILIAS used for a 
CM of health workers. The textbook scenario of a linear testing implementation is compared with concrete results from 
the adaptive testing tool implementation, and the potential for increasing the test efficiency is demonstrated. 


KEYWORDS 

Competence measurement, adaptive testing. Learning Management System, ILIAS, testing efficiency 


1. PROBLEM STATEMENT 

Competence measurement (CM) is a central challenge in competence management. The more exact 
competencies can be measured before a training, the more targeted a training can be (Klett 2010, North 2013, 
Draganidis and Mentzas 2006). A formative or summative assessment following trainings and lectures offers 
the foundation for monitoring learning progress. CMs can vary broadly in their application, ranging from 
large-scale assessments (e.g. the PISA study) to self-testing. 

CMs are characterized by their large scope. There are typically many dimensions to consider that in turn 
depend on numerous characteristics of competency. Erpenbeck and Sauter differentiate various competency 
classes, including personal, activity and decision-making, subject and method, and social-communicative 
(Erpenbeck and Sauter 2013). According to Heyse and Erpenbeck, these can be divided into facets like 
personal responsibility, decision-making ability, analytical or communication skills (Heyse and Erpenbeck 
2007). The measurement task grows with every added dimension. If for example the mathematical and 
linguistic abilities of a test subject are to be assessed, it is usually necessary to pose a distinct task for each. 
Tests cannot however be extended to assess an arbitrary number of areas due to the fatigue of test subjects, to 
the opportunity cost to the examiner (or test subject), or due to testing regulations. A reduction in 
measurement quality or dimensionality must therefore be often accepted in current practice. 

The question arises as to how the efficiency of actual CM test procedures can be improved and in 
particular as to how the measurement quality can be raised for a test of fixed duration. Linear testing 
procedures present an important point for improvement. There are weaknesses in the efficiencies of linear 
tests of participant groups of heterogeneous abilities, as the tests must remain comprehensive to their groups. 
This is reflected, for example, by the use of tasks of varying difficulty. At the same time, participants must be 
presented with the same tasks, thereby being measured at a level that can be either too high or too low. Here, 
computerized adaptive testing promises significant improvements in efficiency by being able to adapt to the 
individual levels of participants as a test progresses. There is however a lack of tools to facilitate integrated 
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adaptive competence measurements. This is the motivation of this paper. At the example of a CM for health 
workers it will be investigated to what extend a computerized adaptive test (CAT) can increase the efficiency 
of a CM. An appropriate tool will be implemented and evaluated. The following research questions follow: 
RQ1: How must a tool for computerized adaptive competence measurement be constructed? 

RQ2: Can a CAT increase the measurement efficiency of a competence assessment? 

Remarks on computerized adaptive testing will follow in section 2. In section 3, the requirements of an 
integrated tool for adaptive competence measurements will be presented. The system choices for the 
implementation of the CM for health workers will be discussed there. The implementation will be presented 
in section 4. The developed tool will be evaluated in section 5. The discussion and interpretation of the 
results comprise the conclusion in section 6. 


2. COMPUTERIZED ADAPTIVE TESTING 

Computerized adaptive testing dates back to the seventies, when powerful computers became increasingly 
available (Lord 1976, Lord 1980, Weiss 1982). Numerous publications have more recently addressed specific 
aspects of adaptive testing, like impacts on participants and motivation (Frey et. al 2009, Tonidandel et. al. 
2002) or content balancing strategies (Zheng et. al. 2013), while others provide comprehensive 
considerations (Van der Linden and Glas 2000). 

The special feature of CATs is that the compilation of their tasks (items), and thus the level of the exam, 
is first established during the test and is dependent on the ongoing performance of the tested person. A more 
difficult task will typically follow a correct answer, and vice versa. For the purposes of this, each item has a 
numeric value representing its degree of difficulty. These values are usually determined in a calibration 
phase, during which the tasks are given to a comprehensive group of test subjects whose performance is 
factored into the statistical models presented below. CATs thus provide an individual testing experience to 
each participant and thereby increase testing efficiency as compared to with classical linear methods, as items 
deemed too easy or difficult can be excluded and each item becomes diagnostically useful. 

CATs aim for the maximum possible performance of the test person. Their goal is to adapt to problems 
for which the examinee has a 50% success rate. Unlike for a linear test, performance cannot be measured by 
the total number of correctly answered questions, due to the varying difficulty of the questions. For this 
reason, CAT is bound by Item Response Theory (IRT), which allows for an assessment of performance based 
on the answered tasks rather than on the test itself (Baker and Kim 2004, Harvey and Hammer 1999). The 
numerical ability parameter (0) reflects the competency level of the participant and replaces the relative 
number of correct answered questions as the test result. The standard error of the test result is calculated in 
real time and provides a measure of the accuracy of the assessment. 

CATs must be separated into branched and tailored types. While for the former a branch of questions is a 
predetermined function of the participant’s answers, for the latter the participant’s testing level is recalculated 
with each answer and the subsequent questions are chosen accordingly from all available tasks in the item 
pool. This offers greater efficiency (Kubinger 2009). 


3. IMPLEMENTATION CRITERIA & SOFTWARE IDENTIFICATION 

The requirements for an IT -based integrated CM tool will be defined in this section. General requirements 
and criteria from the applied sample CM for health workers are mentioned at first. The system selection for 
the CM implementation follows. 

Both open and closed questions are of interest for CMs. Closed questions like multiple-choice facilitate 
an economical assessment of knowledge due to their simple structure. More complex types can address 
functional capacity (e.g. requiring the examinee to select the proper region of a figure). Open questions, like 
free text entries, can assess complex skills (like communication or problem-solving skills e.g. through the 
structuring of answers). The possibility for both open and closed questions is therefore a necessary criterion 
for task creation. Multimedia-based elements can add significant value to a CM by increasing the action of 
the relevant task. They can present complex stimuli (Brunken et. al. 2003 ) that support the situational and 
contextual integration of the tasks and likewise promote the transferability of the test results to real-world 
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situations (Bennett 1999, Jurecka and Hartig 2007, Mayer 2005). The availability of multimedia elements is 
therefore another required criterion. As tailored testing offers the greatest potential for increasing testing 
efficiency, the availability of an adaptive component that supports tailored testing is also required. 

After these general requirements, criteria for the use as testing software for health workers will be 
derived. The vocational training of health workers in Germany follows a dual-study system. The training 
takes place in both a medical practice and a vocational school and requires the completion of an intermediate 
and final examination. The tool should work for both a large-scale summative assessment in the vocational 
school and an assignment in the medical practice, e.g. a formative self-test. Consequently the test tool should 
be highly scalable and applicable under heterogeneous conditions. These requirements could also be applied 
as a typical scenario for many other professions. At the same time, the evaluation must be able to provide 
feedback to the tested persons. The criterion of an intuitive user interface is necessary to minimize barriers 
for test takers not having high levels of computer literacy. As the final examination of the health workers is a 
summative assessment, it is essential for the later evaluation that the results are classifiable. A functionality 
for the archiving of test results is necessary. The encryption of data transfers and secure data storage address 
requirements for the testing security as set by the German federal states. Meeting these requirements ensures 
that the tool could be used for examinations or training purposes in future. 

The test should be split into adaptive and multimedia-based parts, such that the efficiency of the CAT 
could be optimized regarding its layout. Multiple-choice questions without multimedia elements ensure that 
the test remains as simple as possible. For the multimedia-based part, graphics, videos and free text tasks are 
required to increase the activeness. The multimedia-based test will not be discussed here, as this paper 
focusses on the increase to the test efficiency through CAT. 

To identify a suitable software solution, a market analysis was conducted. Following Webster and Watson 
(Webster 2002), a literature search was carried out using the keywords e-assessment, computer-based 
assessment, computerized assessment and computer-based testing, in both English and German. Through the 
search engines Google and Bing, 136 potential software solutions were subsequently identified. 

Since no software could be found satisfying all criteria, focus was turned on finding the most expandable 
solution. All adaptive solutions lacked in several criteria like the opportunity to integrate multimedia 
elements, shortcomings in testing security, and test management. In contrast, six software solutions satisfied 
all criteria except for having an adaptive testing functionality. They were therefore examined more closely. 
The analysis was performed using the categories of issue management, test management, security, 
interoperability, usability, and reliability. 

On this basis, the test component of the learning management system ILIAS (ILIAS 2015, Kunkel 2011) 
was selected for further development. As an open-source software in a standard programming language, 
PHP. ILIAS offered complete freedom for customization and the long-term availability. Being web-based 
and platform-independent, the solution satisfied the requirement for usability under heterogeneous 
conditions. The client server structure allowed for the central storage of all test data. In the event of a system 
crash, tests can be resumed from the appropriate place. There was also high scalability with support for 
extensive user management and numerous simultaneous users. ILIAS offers eleven question types. The 
analysis is largely omitted and allows for summative and formative efforts. 


4. IMPLEMENTATION 

In section 4.1, the concrete requirements for the implementation are described based on the functional criteria 
from section 3. The CM realization is presented in paragraph 4.2. 

4.1 Requirements 

The CAT should diverge as little as possible from other ILIAS functionalities in order to avoid barriers to the 
usability. Lor maximum compatibility, the CAT should work with the same system requirements than 
standard ILIAS (ILIAS 2015, Kunkel 2011). Solely a limited increase in computing power will be necessary 
due to the CAT algorithm. Flowever it should be limited as the test has to be usable under heterogeneous 
conditions in the vocational schools or medical practice. 
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The CAT should be based on Item Response Theory (IRT). By using IRT, it is possible to connect the test 
results with one or more latent variables on an empirical foundation (Baker and Kim 2004, Embretson and 
Reise 2000). In the current case of performance testing, one latent variable is considered to be the 
participants’ ability to solve the test items (0), which is consequently represented as latent trait. The modeling 
of the corresponding representation, which allows the statistical calculation of the latent trait, should be done 
using the Rasch model. It assumes that a person’s ability to solve a test item is based on the persons’ latent 
trait - represented by the estimation of a weighted likelihood estimate - and the difficulty of the item 
- represented by a response model parameter estimate. Both parameters are estimated based on solution 
probabilities and are interdependent in iterations (Bond and Fox 2007, Fischer and Molenaar 1995). 
According to the Rasch model, whether or not a person solves a problem depends only on his or her ability 
and on the difficulty of the task, which are both measured on the same scale. This is a strong assumption, 
with the advantage of facilitating the modeling of the adaptive test solely on these parameters. 

An extensive calibration phase (cp. 5.1) was conducted to determine the difficulty of the tasks and decide 
whether a one- or multi-dimensional competency model should be used. A one-dimensional model was 
selected as multi-dimensional models showed no significant increase in precision. However, the possibility to 
expand the tool later for multi-dimensional CMs should exist. There are various one-dimensional estimation 
methods that implement IRT ( Van der Linden and Glas 2000). Since the tasks were predominantly developed 
from scratch and there existed no prior experience in the calibration of the CAT, the simplicity and 
robustness of the estimation method are a priority, given the complexity of the procedure and the precision of 
the measurements. The expansion for adaptive testing was divided into three parts, namely ( 1 ) the creation of 
tasks, (2) the processing sequence during the test procedure and (3) the storing of the test data and the 
evaluation of the test results. 

For the creation of the adaptive tasks (1), the ILIAS task template must be extended. The difficulty of 
each task needs to be added as numerical value to each question type. A second numerical parameter should 
be created to assign different competency dimensions to the tasks for a potential later multi-dimensional CM. 
Furthermore the possibility should exist to deploy a task as adaptive or non-adaptive in different tests. 
Therefore an additional Boolean parameter should be added. 

The processing sequence during test procedure (2) in case of a CAT must be constructed as the test 
progresses. In contrast ILIAS uses for sequencing tasks an initial test sequence, which is not changed during 
the test. Even though ILIAS offers an option for randomized item selection, only the initial sequence is 
generated randomly in this case. Therefore, a constant reordering of this test sequence during the test has to 
be implemented. There are three determining factors for the adaptive testing procedure. These include 
(a) item selection during the test, (b) item selection at the start of the test, and (c) the conditions for the 
completion of the test (Van der Linden and Glas 2000, Mills and Stocking 1996). All factors depend directly 
on the used estimation method. The expected a posteriori (EAP) approximation of Bock and Mislevy (Bock 
and Mislevy 1982) is chosen, combining decent precision with high robustness and low computing 
requirements. EAP is a Bayesian method. For item selection during the test (a) after each completed task, the 
ability parameter of the participant (0) is estimated and subsequently the most informative unused item for 
the current 0 (= item, whose difficulty is next to (0)) is selected. EAP offers a small bias and standard error 
compared to other Bayesian methods (Wang 1997). The EAP estimates are calculated noniteratively. 
Corresponding calculations using values of the IRT-function can be performed before the beginning of the 
test and stored (Bock and Mislevy 1982), which reduces hardware demands. Another advantage of EAP is 
that it could also be used for the item selection at the start of the test (b), as the approximation is stable over 
the entire test length (Bock and Mislevy 1982). An average skill level (0 = 0) should initially be assumed. 
The standard termination condition (c) in ILIAS for linear tests is met, when the last item in the test sequence 
is passed. For the adaptive test two termination conditions should be established, the first being the static 
completion of a defined number of questions and the second being the dynamic achievement of a certain 
level of precision with regard to the estimate of the standard deviation. 

For the storing of the test data and the evaluation of the test (3), the data storage has to ensure that the 
data remain accessible over the long term. For the evaluation, standard functionality needs to be expanded to 
output the task, the updated measured competency level (0) and standard error following each question. In 
addition, an aggregate index of all questions is required for the evaluation of all test persons and tasks. 
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4.2 CM Realization 

The ILIAS task template was completed for each questions type to establish a level of difficulty, a subject 
area and for the identification of adaptive questions. The user can consequently create or edit adaptive tasks 
through the standard interface without programming skills. 

The CAT algorithm is selected over standard algorithms through a new option in the interface. When a 
question pool is selected by the user for the creation of an adaptive test, all adaptive questions within it are 
used to create an initial test sequence. This ensures that all items are available for the test procedure. During 
the execution of the test, this sequence is continuously refined: After each completed task, the EAP 
estimation is applied to select the next item. The selected item is promoted to the position which is next in the 
test sequence. The standard error is calculated at the same time with 0. At the beginning of the test a value of 
0 = 0 is presumed. A question number limit and a criterion for the standard error in the algorithm are possible 
termination conditions to be used independently or in tandem. The inputs for the termination conditions were 
integrated into the standard interface. When termination conditions are met, the test sequence is shortened to 
this point and the standard termination condition of ILIAS is used to finish the test. 

The test results (0 and standard error) are continuously stored in the central ILIAS database so that the 
CAT can be resumed after a system crash without any loss of progress. The corresponding data and the data 
from all newly created input fields (e. g. difficulty of questions, termination conditions) are archived in a way 
consistent with the other data. Lor the evaluation, two new spreadsheets were created. One table shows the 
individual testing trajectory of the participant, including his or her skill level and the standard error. The 
second gives an overview of all questions posed to at least one user and how they were answered. 

All possibilities for the multimedia design were retained for the CAT. This is a clear benefit in 
comparison to numerous special CAT software solutions. The encapsulated design of the testing tool 
facilitates the integration of additional functionalities, e.g. multi-dimensional estimation procedures. 


5. EVALUATION 

This section presents the evaluation of the testing instrument. The tool was calibrated and evaluated in a large 
scale scenario with real scholars in the vocational schools. A description of the general experience 
concerning the application is given first. A comparison of concrete measurements with those of a typical 
linear test is found afterwards. Lastly the specific use of the developed testing tool is addressed. 

5.1 Quality Assurance and Experience with the Implementation 

The newly developed tasks were tested in two ways. In a first calibration stage in order to establish the 
degree of difficulty of each task they were presented linearly to approximately 1.200 health worker scholars 
in the vocational schools. The graduating class was chosen for maximum comparability of the results for the 
final examination. Lurthermore they were examined in expert workshops. The usability of the test proved to 
be very good. Since the surface of the CAT did not differ from a linear standard ILIAS test, a bias due to a 
different interface could be excluded. The items were coded dichotomous. Partially correct solutions were 
evaluated as false in order to increase their difficulty. After final revisions, scaling and testing of Rasch 
conformity, 88 items covering a skill range (0) of -2,091 to 2,678 remained. 

In a second stage these items were used in main survey for a span of 15 to 30 questions. As no detailed 
experiences regarding the expected precision were available, a fixed number of questions was chosen in favor 
of a variable test length. A total of 1.183 data sets were collected through adaptive testing. These data sets 
were not simulated, but measured using participants that differed from the first stage. 

5.2 Comparison of the Linear and Adaptive Testing Procedures 

To check, whether the adaptive design of the tests could increase the measurement efficiency, a textbook 
example of a linear test procedure was created as a basis for comparison. Live people with a skill level 
spanning from +2 (high) to -2 (low) were taken with a test length of 15 questions equally divided into five 
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difficulty levels. While the specification of skill levels in CATs depends on the statistical model, it is 
specified here manually to provide integer number levels of competency. The scale is interpolated to the 
value range of the CAT for comparison. The textbook example is shown in table 1. 


Table 1. Linear testing order 


task number 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13 

14 

15 

0 test taker = 

no contribution 

0 task 

-2 

_2 

-2 

-1 

-1 

-1 

0 

0 

0 

1 

1 

1 

2 

2 

2 

6 task 

to measurement 

test taker: 0 = +2 

<0 

<0 

<0 

<0 

<0 

<0 

<0 

<0 

<0 

<0 

<0 

<0 

= 0 

= 0 

= 0 

20,0% 

80,0% 

test taker: 0 = +l 

<0 

<e 

<0 

<0 

<0 

<0 

<0 

<0 

<0 

= e 

= 0 

= 6 

>0 

>6 

>0 

20,0% 

60,0% 

test taker: 0=0 

<0 

<0 

<0 

<0 

<0 

<0 

= 0 

= 0 

= 0 

>0 

>0 

>0 

>0 

>0 

>0 

20,0% 

60,0% 

test taker: 0 = -1 

<e 

<e 

<0 

= 0 

= 0 

= 0 

>0 

>0 

>0 

>0 

>0 

>0 

>0 

>0 

>0 

20,0% 

60,0% 

test taker: 0 = -2 

= e 

= 0 

= 0 

>0 

>0 

>0 

>0 

>0 

>0 

>0 

>0 

>0 

>0 

>0 

>0 

20,0% 

60,0% 


The number of individual test questions is presented in the first line, the difficulty level of the tasks in the 
second, and the results for the individual test persons in the five following lines. The tests were completed 
from left to right. It is assumed that the difficulty of the tasks is the only influencing factor, such that a 
respondent properly completes all tasks of the same or lower skill level (test person 0 >= task 0). No 
distinctions are made according to the content areas. A dark cell marking signifies that a task has the same 
skill level as the test person, while a light one signifies that it was taken as a bound. The last two columns 
present the portion of tasks at the skill level of the test person and the measurements of irrelevant tasks (no 
marking). 

For all five sample test persons, the individual skill level was only matched for 20% of the tasks. The 
strongest test person (0 = 2) initially needed twelve items below his or her skill level before receiving items 
13 to 15 at this level. For the weakest test person (0 = -2), only the first three tasks were on the appropriate 
level and questions 4 to 15 were too difficult. The other cases performed equivalently, with only the position 
of the task fitting the test person’s skill level changing. Except for the strongest test person, at least one 
further task was necessary to delineate the upper limit of the skill level. It could be generously argued that the 
entire next level of difficulty is relevant to distinguishing the skill limit (light marking), such that 20% to 
40% of the questions add value to the measurement. In reverse, for the present example 60% to 80% of the 
questions (no marking) have no direct contribution to the measurement, as they are too easy or difficult. This 
rate could be reduced with the difficulty level value. Apart from the loss of precision, however, the basic 
problem would persist. 

To test whether improvements can be achieved through the CAT for this idealized scenario, the scenario 
was adopted and supported by measured values from the CAT application. Based on the stepped skill levels 
+2, +1, 0, -1 and -2, one of the 1.183 measured data sets was chosen for each test person based on which 
most nearly fit his or her measured 0 as evaluated after 15 questions. Test persons with 0 values of +1.95, 
+ 1.00, 0.00, -0.99, and -1.97 were thereby identified. Table 2 illustrates the testing procedure for these test 
persons, showing the measured skill values following each question. 


Table 2. Adaptive testing order 


Task Number 

i 

2 

3 

4 

5 

6 

7 

8 

1 

9 j 10 

ii 

12 

13 

14 

15 

0 test taker = 
0 task 

no contribution 
to measurement 

test taker 0 = + 1,95 
(+1,5 to +2,5) 

0,41 

0.76 

1.07 

1.34 

1.59 

1.82 

2,04 

2.20 

2.05 j 2,16 

1 

1.91 

2.00 

1,80 

1,88 

1.95 

73,3% 

0,0% 

test taker 0 =+1,00 
(+0,5 to +1,5) 

0,41 

0.06 

- 0,24 

0,02 

0,26 

0.47 

0,67 

0,85 

1 

1.031 0,86 

1.01 

0.87 

1,01 

0,88 

1.00 

60,0% 

0,0% 

test taker 0 = 0,00 
(-0,5 to +0,5) 

- 0.41 

- 0.06 

0.24 

- 0.03 

- 0.26 

- 0,05 

- 0.24 

- 0.42 

1 

- 0, 25 j - 0,40 

i 

- 0.26 

- 0,13 

- 0.01 

0,11 

0.00 

100,0% 

0,0% 

test taker 0 = - 0,99 
(-1,5 to -0,5) 

- 0.41 

- 0.06 

0.24 

- 0,03 

- 0.26 

- 0,47 

- 0.67 

- 0.85 

i 

- 0 , 681 - 0,83 

1 

- 0.98 

- 1,12 

- 0,99 

- 1.11 

- 0,99 

93,3% 

0,0% 

test taker 0 = - 1,97 
(-2,5 to -1,5) 

- 0.41 

- 0.76 

- 1,07 

- 1.34 

- 1.59 

- 1,82 

- 1.60 

- 1.78 

1 

- l , 63 j - 1,77 

- 1.89 

- 2,00 

- 2.09 

- 1.90 

- 1.97 

73,3% 

0,0% 


47 


ISBN: 978-989-8533-40-1 ©2015 


The presentation follows the likeness of the previous figure. Each participant answered 15 questions, 
numbered left to right. Due to the adaptive nature of the test, the questions differed between participants. As 
there can be no universal statement of the difficulty of the tasks, the second line is omitted. For each question 
number, the absolute measurement result (= test person’s 0, to two decimal places) following each question is 
presented. Questions at the skill level of the test persons are again characterized by dark cell markers. Light 
highlights are used to distinguish contributing questions. To determine whether a question was at the skill 
level of a test person, an interval of 0 +/- 0.5 was taken as a basis. This would in the worst case correspond to 
the same accuracy limit of the linear textbook scenario: For the textbook example, the difficulty level must 
exceed the participant’s skill level by 1 (e.g. to 0 = 2 for a participant with 0 = 1), whereas for the CAT an 
interval extending both above and below the participant’s skill level is necessary (e.g. the interval 0 = 0.5 to 
0 = 1.5 for a participant with 0=1). 

It is apparent that between 60 and 100% of the questions addressed the relevant skill level. The share of 
too easy or difficult questions is reduced to 0 - 40%. The portion of questions not affecting the measurement 
is also reduced to 0%, since each question served part of the algorithm branch and the determination of 0. As 
expected, the border values fall weakly for high and low 0, since the algorithm takes longer to settle on the 
appropriate skill level. It becomes clear, however, that a skill level within the appropriate interval is reached 
after seven consecutive questions and that a nearly constant 0 is reached after nine questions. With question 
nine, each test person was given three tasks at his or her own skill level, like in the linear test. This is 
delineated by the vertical dashed line in table 2. On average, each test person was already posed five 
questions up to this point at his or her own skill level. The black vertical line in table 2 marks the point at 
which each test person was posed the third question at his or her own skill level. For the presented example, 
at least 60% of the questions (after question 9 instead of 15) would have already been reached at this point. 
Figure 1 underlines this effect. 



The questions are shown on the x-axis, the measured 0 on the y-axis. Similar to in previous figures, 0 is 
displayed after the execution of each question, such that a nonzero value already follows question 1. 
Horizontal lines indicate the different intervals. The desired increase in the test efficiency due to the CAT 
algorithm could be confirmed by the example case. The mean value of 0 was calculated over all 1.183 
collected data sets after questions 9 and 15. Both values were very similar (0.261 versus 0.296). At the same 
time, the positive value of 0 shows that the CAT was slightly easier than expected. 

With regard to the items, the question arises as to how they were used in the test. Figure 2 shows which 
item was used in which position for all 88 tasks based on all data sets. 



Figure 2. Item frequency and item position in test 
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For a test length of 15 questions, the positions within the tests are shown horizontally and the 88 items are 
shown vertically according to their difficulty levels, from difficult (item 1) to easy (item 88). The more test 
persons who were posed an item at a specific location, the darker the corresponding cell is shaded. (The 
darkest shading corresponds to more than 40% of the test persons being posed the item at this position, dark 
shading corresponds to 8-40%, light to 2-8%, and the lightest to 0-2%). The last column shows how many 
test persons an item was posed to in total. 

It becomes clear that all test persons were given the same item with intermediate difficulty at the start of 
the test (see column 1 ). This conforms to our expectations, since all test persons have the same 0 of 0 at the 
beginning of the test. Dependent on the correct or incorrect answering of this question, an easier or more 
difficult subsequent item is selected (column 2). Based on these two items, the algorithm branches to four 
items, then eight and sixteen. The branches may overlap for the first time with the sixth question and 30 
items (column 6). By the ninth item (column 9) it becomes clear that all questions, spanning the entire 
difficulty spectrum, not posed up to this point belong to parallel paths. This shows how fine grained the 
measurement already is up to this point. It should be noted that the more difficult items were more frequently 
used, confirming the test was easier than expected. It also becomes clear that the frequency of the use of 
items of intermediate difficulty decreased towards the limits of higher or lower difficulty. This is expected, as 
the central difficulty range is more frequented than the edge ranges of higher and lower difficulty. 


6. CONCLUSION 

The central point of this contribution was to examine whether CAT could improve testing efficiency in the 
example of a medical competency measurement, as well as to present the relevant implementation and 
functional ability of an CAT component in ILIAS. 

The results show that a fully valued CAT was successfully developed for the learning management 
system ILIAS. The test reacts adaptively to user inputs and selects the necessary subsequent tasks during 
runtime. The requirements in section 3 and the implementation in section 4 provide a detailed answer to 
research question 1, how to construct a computerized tool for adaptive competence assessment. Research 
question 2, focusing on the increase in measurement efficiency for a competence assessment, was addressed 
in section 5. A clear increase in the measurement efficiency could be achieved for the presented 
implementation case of a CM for health workers. As compared to a traditional testing format, the test time 
was reduced by 40%. The test persons were already posed three questions addressing their competency level 
after 9 questions in total, as compared to after 15 total questions for a linear test procedure. 

In summary, an integrated tool was created for the competency measurement, with which a multifaceted 
adaptive competency measurement can be created from comprehensive types of questions and multimedia 
elements. No other tool could be identified possessing these capabilities, including extensive reporting 
options, graphical interface and high test security. This facilitates the potential for future implementations of 
combined multimedia-CATs. The tool supports large-scale testing and summative, diagnostic or formative 
usage. The time savings realized through the implementation of the CAT can be utilized as part of an 
integrated competency measurement with further testing. In practice measurement quality or dimensionality 
of the test can be improved for the same participants or more participants could be tested within the same 
time. The mapping of complex action situations in multimedia tests could replace personnel-intensive oral 
examinations. This benefit is not limited to CMs in the medical field. 

As a contribution to knowledge, along with the time savings, the testing tool could facilitate more detailed 
and larger-scale competency measurements. It could enable large-scale empirical studies on the interaction 
between competency dimensions that are currently not feasible because of the associated expenses. 
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ABSTRACT 

The paper introduces a virtual collaborative learning setting called ‘Net Economy’, which we established as part of an 
international learning network of currently seven universities. Using the Community of Inquiry framework as guidance 
and Canonical Action Research (CAR) as the chosen research design, the discussion forum of the online course is 
assessed regarding its critical thinking value. We thereby measure critical thinking with the help of the according model 
provided by Newman et al. (1995), which differentiates 40 indicators of critical thinking from ten different categories. 
The calculated critical thinking ratios for the analyzed two discussion threads indicate a strong use of outside knowledge, 
intensive justification as well as critical assessment of posts by the students. But at the same time there are also weak 
spots, like manifold repetitions. Based on these results we derive changes for the next course cycle in order to improve 
the critical thinking of the students. 


KEYWORDS 

Virtual Collaborative Learning, Critical Thinking, Community of Inquiry. Forum Analysis 


1. INTRODUCTION 

Learning with new media is becoming increasingly popular in times of MOOCs and a continuing growth in 
the use of mobile devices. Outside the traditional classroom, learning can spread to more diverse settings, 
contexts, and locations. As at the same time virtual collaboration is gaining significance in business, virtual 
collaborative learning (VLC) as a sophisticated type of eLearning setting gains growing attention in both 
research and higher education practice. The accompanying pedagogical and technological innovations 
thereby create new challenges for the instructors and also redefine their roles. 

Successful eLearning requires learning environments that motivate the students and facilitate meaningful 
and worthwhile learning. A suitable design and a well -structured learning process need to consider social, 
technical and didactic challenges as interdependent dimensions of a good learning experience 
(Garrison 2011). VCL-settings thereby aim at the skills that are required and considered indispensable for 
virtual collaboration, involving team members working from various international locations and with 
heterogeneous cultural and educational backgrounds. 

This paper introduces our own VCL learning scenario ‘Net Economy’ as part of a continuous 
build-and-evaluate loop, while our improvement efforts follow the iterative steps of canonical action research 
(CAR): diagnosis, action planning, intervention, evaluation, and reflection (Davison et al. 2004; Susman & 
Evered 1978). Action research is committed to “[...] the production of new knowledge through the seeking 
of solutions or improvements to ‘real-life’ problem situations.” (McKay & Marshall 2001) It thus serves 
both, research and practice by studying the real world while considering a particular theoretical framework. 

For our eLearning action research the Community of Inquiry (Col) as introduced by Garrison et al. (2000) 
serves as the required theoretical framework, consisting of the three interdependent elements of 1) social 
presence, 2) cognitive presence and 3) teaching presence. While social presence is “the ability of participants 
to identify with a group, communicate purposefully in a trusting environment, and develop personal and 
affective relationships progressively by way of projecting their individual personalities” 
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(Garrison 2011, p. 23), cognitive presence is “the extent to which learners are able to construct and confirm 
meaning through sustained reflection and discourse in a critical community of inquiry” (Garrison et al. 2001 ). 
Teaching presence, finally, is “the design, facilitation, and direction of cognitive and social processes for the 
purpose of realizing personally meaningful and educationally worthwhile learning outcomes” and thus 
merges all elements in a balanced and functional relationship. 

While in previous papers we developed solutions mainly for various problems of teaching and social 
presence, like e.g. the use of a dedicated social network instead of a learning management system, we have so 
far not explicitly addressed the eLearning elements of the setting with regard to the required cognitive 
presence. According to the Col framework students need to be “engaged in a collaborative and reflective 
process which includes understanding an issue or problem, searching for relevant information, connecting 
and integrating information, and actively confirming the understanding” (Garrison 2011, p. 94). An obvious 
instrument to do so seem to be discussion forums, which are therefore commonly used in eLearning. 
Computer-mediated communication tools, like discussion forums, provide ways for learners to interact. 
Research suggests that asynchronous online discussions are even superior with regard to critical thinking and 
knowledge co-construction due to the available time for reflection and more critical and constructive 
contributions when compared to synchronous face-to-face discussions (Wang et al. 2009). On the other hand, 
in practice, it usually stays unassessed and questionable whether the initiated discussions really provided for 
the intended critical and reflective interaction among the participants and thus for cognitive presence. 

In the following we, therefore, measure and analyze the critical thinking ratio of the discussion forum that 
we established in our ‘Net Economy’ setting as part of a required discussion task. Based on this diagnosis 
step we will then derive interventions that aim at improving the critical and reflective thinking and thus the 
cognitive presence. After a short introduction of our VCL-setting in section 2, we will outline the diagnosis 
approach as adapted from Newman et al. (1995) and assess the critical thinking in our implemented 
discussion forum in section 3. In section 4 we will then derive interventions as improvement suggestions for 
the next course cycle before we conclude in section 5. 


2. THE NET ECONOMY VCL-SETTING 

‘Net Economy’ is an annual cross-location virtual collaborative learning setting (VCL) in the framework of 
an international learning network which we established in 2008. The setting targets participants with 
heterogeneous educational backgrounds in the fields of business and economics, business informatics, 
international economy and hotel & tourism management, as well as different cultural backgrounds from 
Germany (Berlin, Mtihlheim an der Ruhr, Soest), Indonesia (Jakarta), Crimea (Simferopol) and Sweden 
(Kristianstad). The course is taught in English and deals with entrepreneurship and business models in the net 
economy. Between 80 and 150 students took part in each course cycle so far and worked in virtual 
intercultural teams on exercises concerning eBusiness models and virtual collaboration. All partners agreed 
on the general learning scenario, leading to a stable VCL-concept which we systematically improve using the 
previously mentioned canonical action research approach. 

‘Net Economy’ is a complete online class with only three parallel classroom meetings at the various 
locations at the beginning of the different phases of the course, as depicted in fig. 1 . A closed social network 
(established with Wordpress and several plugins) serves as the major coordination platform for the course 
and facilitates social presence. Every student needs to set up a profile and join a team within the first weeks 
of the class, called Preparation Phase. During these weeks, the whole concept is introduced and the different 
tasks and roles within a team are described and assigned. The students team up to groups of five, in which 
every team member has a very specific role to fulfill. Only two members of the same university are allowed 
to be part of the same group so that a high diversity is given in each team. In the following Knowledge 
Development Phase, the students are then introduced to some major characteristics, concepts and challenges 
of eCommerce and the Web 2.0 and to major types of business models on the web. On each of the topics 
introduced by the instructors via eLecture, the individual students as well as the teams have to complete and 
submit assignments. Each participating location is responsible for one of these eLectures and an according 
online assignment. Knowledge Development Phase, ends with a multiple choice test on the topics covered in 
the eLectures. In the final Case Study Phase every team is then asked to apply the acquired knowledge and 
develop a business plan for an assigned business model type using the so-called business model canvas. This 
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Case Study Phase includes a peer review process in which each team receives feedback from at least two 
other teams. The top rated teams then present their business plan during a final live online meeting, in which 
all participants will vote for the top E-Venture of the class. 



Bookmarks 
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Social 


Tag 

Clouds 


fell E-Electures [c] Classroom Presentation (int'l Online-Meeting) □ Online Phase (group work phase) 


Figure 1. The Net Economy Learning Scenario 

In the last course cycle accomplished between October 2014 and February 2015, students were asked to 
contribute to an online discussion individually as a first assignment. They could choose between two 
discussion threads on which they got a reading and had to write a first forum post pointing out their personal 
opinion and thoughts regarding the topics of the articles. In addition, they had to comment on at least two 
other posts and emphasize in their statements whether they support or rebut the post they are answering to, in 
order to facilitate critical and reflective thinking. Discussion thread 1 discussed the online available WIRED 
article “The Web is dead. Long live the Internet.” by Chris Anderson and Michael Wolff and initiated 
discussion with the question: “How does the article and the discussion around it reflect today’s eCommerce 
trends and what consequences would you expect from the development for popular online businesses?” 
Discussion thread 2 dealt with the as well online available Fast Company article “Encyclopaedia Britannica 
(EB) is dead, long live Encyclopaedia Britannica”, raising the question: “Why did the original business 
model have to change and how did the Encyclopedia Britannica adapt? Do you believe in the viability of the 
business model?” The not any further moderated forum aimed to facilitate discourse and make the students 
think critically about the provided articles. 

In total 80 members were registered on the ‘Net Economy’ network in the last course cycle. Next to the 
lecturers 13 teams of five active students, in total 65 students, were working on the platform. 56 students 
(86%) participated in this first individual assignment and 166 replies were posted, which are about three 
posts per participating student, indicating that the students fulfilled their task of writing one statement and 
two replies. To investigate the qualitative value of the forum posts, however, a closer look has to be taken at 
the posts themselves and a more qualitative analysis is required, as accomplished in the following chapters. 


3. METHODOLOGY & APPLICATION OL METHOD 
3.1 Critical Thinking 

In the literature, many definitions of critical thinking can be found. According to the Foundation for Critical 
Thinking (2015) the word “critical” derives etymologically from two Greek roots: “kriticos” (meaning 
discerning judgment) and "kriterion" (meaning standards). Etymologically, then, the word implies the 
development of “discerning judgment based on standards.” Ennis (1993) defines critical thinking as 
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“reasonable reflective thinking focused on deciding what to believe or do.” In his opinion, a person needs to 
e.g. judge the credibility of sources and the quality of arguments, identify conclusions, reasons, and 
assumptions or develop and defend a position on an issue. Critical thinking therefore requires a wide set of 
information, generating processes and standards to base one’s thinking on and relies on the person involved 
to have the intellectual commitment to make use of these skills and standards to guide his/her behavior. 

Thus, critical thinking is a state of thinking that is beyond one’s own, or even groups, interests and is 
dependent on the quality of standards and depth of experience the thinker has in respect to a particular 
problem or question. It also depends on different values and cultures, as differences in culture can have 
different effects and interpretations of being critical, as “in some cultures, being critical maybe interpreted as 
‘argumentative’ or ‘being critical of others’” (Woo & Wang 2009). Simplified one can say that critical 
thinking involves seeking information, analyzing alternatives, evaluating the alternatives in relation to your 
aims and requirements and reaching a conclusion to the problem or answer. 

In addition, critical thinking can also relate to other important areas in learning. Lai (2011) for example, 
says that “critical thinking skills relate to several other important student learning outcomes, such as 
metacognition, motivation, collaboration, and creativity.” Being able to assess one’s own arguments and 
reasoning is necessary for self-regulated learning. Tasks, which spark interest and are challenging the 
students usually call for critical thinking and trigger motivation. Collaboration requires the students to think 
in diverse perspectives and the more diverse they can think the better collaborators they may become. 
Opportunities for collaboration may also encourage higher quality thinking itself. Finally, also creative 
thinking incorporates attributes of critical thinking, such as open-mindedness and flexibility. All these 
attributes enhance the learning experience for the students and stimulate intellectual and personal growth 
(Lai 2011). 

3.2 Assessing Critical Thinking through Content Analysis 

In literature, many instruments are described which focus on content analysis and aim at providing evidence 
of learning and the knowledge construction taking place. These instruments represent a wide variety of 
approaches and differ in their level of detail, type of analysis categories and most important the diversity of 
their theoretical base (Wever et al. 2006). Content analysis instruments need to be objective, reliable, 
replicable, and systematic (Rourke et al. 2001). Instruments to measure critical thinking in terms of content 
analysis in a thematic unit of analysis were for example introduced by Newman et al. (1995) and Bullen 
(1998). 

Whereas Bullen's framework is based on different conceptualizations of critical thinking and consists of 
four different categories of critical thinking skills, Newman et al.'s approach is based on Garrison's (1991) 
five stages of critical thinking and Henri's (1992; cited in: Wever et al. 2006) cognitive skills. In his papers, 
Garrison describes critical thinking as a problem-solving process in which the critical thinker will move 
through five stages: problem identification, problem definition, problem exploration, problem applicability, 
problem integration. Marra et al. (2004) evaluate the Newman et al. protocol to be good for high-level 
descriptive data of what is happening in a discussion and clearly defined with numerous codes and coding 
rules. In our study we, therefore, used this approach to analyze the critical thinking of our students in the 
above-described discussion scenario. 

A prerequisite for this method, however, is that critical thinking indicators are identifiable, or in other 
words “critical thinking should be defined in such a way that it is measurable" (Woo & Wang 2009). In order 
to cover all possible types of posts in a discussion forum, Newman et al. (1995) expanded the five stages into 
the following 10 categories of critical thinking indicators: Relevance, Importance, Novelty, Ambiguity, 
Outside Knowledge, Linking ideas, Justification, Critical Assessment, Practical Utility and Width of 
Understanding. Each of the indicators has a pair of opposites, one for in-depth processing, and one for 
surface learning (Newman et al. 1995). Newman et al. have developed a full set of 40 indicators within the 
ten categories, as depicted in fig. 2. 
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Category 


Positive Indicator 

Negative Indicator 

R± 

Relevance 

R+ 

Relevant statements 

R- Irrelevant statements, diversions 

I± 

Importance 

1+ 

Important points / issues 

I- Unimportant, trivial points /issues 

N± 

Novelty; new info, ideas. 

NP+ 

New problem-related 

NP- Repeating what has been said 


solutions 


information 



NI+ 

New ideas for discussion 

NI- False or trivial leads 



NS+ 

New solutions to problems 

NS- Accepting first offered solution 



NQ+ 

Welcoming new ideas 

NQ- Squashing, putting down new ideas 


Figure 2. Excerpt of coding indicators of the Newman et al. framework (Marra et al. 2004) 


While the categories are provided in the left column, the individual indicators and their pairs with the 
matching codes follow in the second and third column. A positive indicator for the Novelty category could, 
for example, be new problem-related information (NP+) provided by a student or the welcoming of new 
ideas (NQ+). Negative indicators would, for instance, be repetitions (NP-), or even the dismissal of new 
ideas of a previous speaker (NQ-) as well as irrelevant statements (R-). 

Along these provided categories with their various indicators, every post needs to be evaluated separately 
and codified accordingly. Statements may cover phrases, sentences, paragraphs or mere messages containing 
one unit of meaning and referring to least one of the indicators. It may even happen that some indicators 
overlap with each other (see figure 3 for example). This is why the coding needs to be carried out in 
chronological order, evaluating the first post first and the later posts to the end of the evaluation sequence. 
Only in this manner, the posts can be assessed properly and new content or repetitions can be identified 
correctly. Furthermore, the links between the different individual threads can be identified this way. But 
rather than classifying and coding every single word it is reasonable and even suggested to mark and count 
the obvious examples, and ignore the intermediate shades of gray (Newman et al. 1995). 

After assessing all posts in this way the approach allows for the calculation of a critical thinking ratio 
based on formula 1 depicted below. The x ratio with x identifying the category is calculated by subtracting 
the sum of all negative indicator from the sum of all positive indicators, divided by the sum off all indicators. 
The resulting measure thereby expresses solely the quality of the text without depending on the quantity of 
participants. This way ten critical thinking ratios are calculated, one for each category, and all of them in a 
range of —1 (all uncritical, all surface) to +1 (all critical, all surface) (Newman et al. 1995). 

x + — x~ 

x ratio = t — 

x ± 

Formula 1. Calculation of critical thinking ratio (simplified according to Newman et al. 1995) 

3.3 Forum Analysis 

The following analysis of the above-explained discussion forum was conducted with the help of the students 
of the ‘Net Economy’ course themselves. One student of each team was assigned the role of a research 
partner and as such had to mark the transcripts of the discussion thread that he did not participate in himself. 
The research partners were introduced to the job and the approach in a web conference, and, in addition, a 
short tutorial was provided on how the free coding software QDAminer Lite can be used to do the job. The 
transcripts of the forums were provided to the students together with the indicators as a predefined project in 
QDAminer Lite. Nine out of the 13 research partners submitted complete and acceptable coding projects with 
four students covering discussion thread 1 and five students covering thread 2. 

The analysis to follow is a merger of these nine complete coding projects provided by the students. Fig. 3 
shows how the coding of a post looks like in QDAminer with the positive and the negative indicators being 
depicted in the right. Building on to these individual codings we then proceeded by summing up all 
appearances of the different indicators and by calculating the critical thinking ratio for each category. While 
the two discussion threads were coded separately we later also merged the results, since both threads were set 
up and managed in the exact same way. This way we were able to calculate critical thinking ratios also for 
the discussion forum as a whole. 


55 



ISBN: 978-989-8533-40-1 ©2015 


The article „The web is dead" clarifies that the wide open web browser will be more and more replaced 



by more convenient semiclosed Apps, which are created for the operating systems of mobile phones. 


R+ 

Apps use the internet to transfer the information to the customer after he had downloaded the 


I- 

application. There are different types of Apps starting from extreme utility to entertainment. Tor the 



users of tablet computers and smartphones it is the most comfortable and fastest way to get to the 


NP- 

desired information. Another Reason for the increase of the numbers of Apps is our fast moving time 



which causes that people do not have enough time to use the well known web browser. This move is 


JS+ 


Figure 3. Coding of forum posts 

Several problems accompanying the chosen analysis approach and research design need to be considered. 
Next to the apple-and-pear problem because of possible variances between the different codings of the 
students, also very small numbers of appearances of specific indicators lead to the risk of over-interpretation. 
We, therefore, do not consider the critical thinking ratios of the Practical Utility and the Width of 
Understanding category in our analysis, as these indicators were used too rarely. Another limitation to keep 
in mind is that the chosen critical thinking model does not answer any specific question or solve any specific 
problem, but rather develops an overall rating for the quality of the interactions between the students and 
their discussion behavior. 


4. RESULTS & ACTION PLANNING 

After merging the coding projects, the results shown in table 1 can be calculated for the ten categories. We 
were able to evaluate eight of the ten categories when considering the sample sizes. Only for the Ambiguity 
category a negative result was calculated, indicating that the coding students found many confusing 
statements. All other ratios are positive with mostly rather small differences between the ratios for the 
discussion threads 1 and 2. As Newman et al. do not provide a scheme for interpreting the results except 
for-1 indicating all uncritical and surface posts and +1 indicating all critical and deep contributions, there is a 
clear need to compare the derived ratios with the ratios of comparable settings. This actually matches our 
CAR approach of a continuous build-and-develop loop, since we will derive interventions in the next step 
that aim at improving the critical thinking in the implemented discussion forum, with the critical thinking 
ratios allowing us to assess and evaluate any achieved improvement. 


Table 1. Critical thinking ratios in the ‘Net Economy’ forum calculated after Newman et al. (1995) 





Topic 1 


Topic 2 


Combined 

Category 

Indicator 

Ratio 

Indicator 

Ratio 

Indicator 

Ratio 



+ 

- 


+ 

- 


+ 

- 


R+ 

Relevance 

118 

46 

0,44 

125 

85 

0,19 

243 

131 

0,30 

I± 

Importance 

58 

53 

0,05 

34 

11 

0,51 

92 

64 

0,18 

N+ 

Novelty 

201 

156 

0,13 

158 

75 

0,36 

359 

231 

0,22 

A+ 

Ambiguity 

35 

72 

-0,35 

19 

42 

-0,38 

54 

114 

-0,36 

o+ 

Outside Knowledge 

189 

27 

0,75 

102 

14 

0,76 

291 

41 

0,75 

L± 

Linking Ideas 

62 

41 

0,20 

29 

31 

-0,03 

91 

72 

0,12 

J± 

Justification 

184 

20 

0,80 

137 

31 

0,63 

321 

51 

0,73 

C± 

Critical Assessment 

55 

12 

0,64 

74 

6 

0,85 

129 

18 

0,76 

P+ 

Practical Utility * 

10 

0 

*0) 

5 

5 

0 

15 

5 

* (0,50) 

w± 

Width of Understanding * 

15 

2 

* (0,76) 

12 

7 

0,26 

27 

9 

* (0,50) 


* These categories are not considered any further due to too small samples. 


The high scores (CT>0,70) among the above results nevertheless point at several positive characteristics 
of the students discussion behavior. Outside knowledge was used a lot (CT=0,75), justifications were 
regularly provided (CT=0,73) and the posts were mostly critically assessed by fellow students (CT=0,76). A 
drill-down into one of the higher and one of the lower scores provides a clearer view of the assessed 
discussion. Table 2 adds an overview of the samples of single indicators from the two categories Linking 
Ideas and Outside Knowledge. Personal experience and previous knowledge, as well as additional course 
material and external resources, were introduced frequently by the students. Furthermore, they were also able 
to link facts and ideas, but thereby continuously repeated information and statements provided by other 
students before. 
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Table 2. Closer Look on the Outside Knowledge and Linking Ideas category and the according numbers 


Category 

+ Indicator 

Count 

- Indicator 

Count 

0+ bringing outside 

OE+ 

Drawing on 

129 

OQ 

Squashing attempts to bring in 

6 

knowledge / 


personal experience 



outside knowledge 


experience to bear on 

OC+ 

Refer to course 

54 

0 

Sticking to prejudice or 

22 

problem 


material 



assumptions 


OM+ 

Use relevant outside 
material 

60 






OK+ 

Using previous 
knowledge 

41 





OP+ 

Course related 
problems brought in 

19 





OQ+ 

Welcoming outside 
knowledge 

1 




L± linking ideas, 

L+ 

Linking facts, ideas 

81 

L- 

Repeating information without 

34 

interpretation 


and notions 



making inferences or offering an 
interpretation 



L+ 

Generating new 

10 

L- 

Stating that one shares the ideas 

38 



data from 



or opinions stated, without taking 




information 



these further or adding any 




collected 



personal comments. 



While these results indicate an overall positive discussion behavior from our point of view, the total 
number of posts points at a rather tightly focused fulfillment of the discussion task (one post + two replies). 
Students seem not to have participated in the discussion based on pure interest, curiosity or fun, but rather 
because it was a required task. The chosen topics and the way of raising and structuring the discussion seems 
not to have managed to turn the students’ extrinsic motivation into intrinsic motivation, which should have 
resulted in some sort of over-fulfillment of the discussion task. The identified numerous repetitions of 
information support this interpretation. 

As it is our goal to foster critical and reflective thinking and thus cognitive presence, we aim at engaging 
the students’ in a more intrinsically motivated discussion in the future. Considering the above results we 
derived the following three interventions regarding the implementation of the discussion forum for the 
upcoming course cycle: 1) More polarizing discussion starters will be developed, including other media 
types, like e.g. video statements, pictures or podcasts in order address the students media behavior more 
systematically. 2) The instructors will contribute to the discussion with prepared statements that open up new 
directions of thinking from time to time so that the students get new ideas and notice that their posts are not 
only counted but read. 3) Discussion forums with consistent requirements and settings will be used in several 
phases of the course and critical and reflective thinking as a desired learning outcome will be turned into a 
subject of discussion itself in order to evolve a discussion culture. 

After the next course cycle, we will assess the discussion forums and calculate the according critical 
thinking rations again in order to evaluate our interventions. This way we hope to have started an 
improvement process for the discussion forums as a major instrument of cognitive presence in eLearning. 


5. CONCLUSION, DISCUSSION & FUTURE WORK 

In this paper, we introduced the VCL-course ‘Net Economy’ and our iterative Canonical Action Research 
(CAR) approach aiming at a systematic improvement of the setting. Using the Community of Inquiry 
framework (Col) as guidance, we focused on cognitive presence as one major requirement of successful 
online learning, which we intended to provide for by implementing a discussion forum and an according 
discussion assignment among other tasks. We evaluated the discussion forum regarding its cognitive value by 
calculating the critical thinking ratios for all ten categories suggested by the Newman et al. (1995) critical 
thinking diagnosis model. Using this approach we were able to diagnose some strength and weaknesses of 
our discussion setting and derived interventions to be implemented in the upcoming course cycle. 

The Newman et al. model was simple to use and user-friendly since all categories and indicators are 
predefined. It provided for quality scores indicating the cognitive value of our discussion forum while not 
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depending on the number of participants. But despite this well-organized and user-friendly model, it turned 
out to be quite tedious and time-consuming to work through the assessed forum with regard to all 40 
indicators contained in the model. A problematic aspect is also the interpretation of the derived results. While 
it is clear that the more the values tend towards -1 or +1 the worse respectively better discussion behavior 
they indicate, there is no guidance as where a good value might start or a bad value might end. 

Regarding the analyzed discussion forum we found that the students frequently introduced outside 
knowledge to the discussion, that they argued intensely, and that they thereby addressed posts critically. On 
the other hand, they often repeated information and posted a high number of rather confusing statements. 
However, the various cultural and educational backgrounds of the students might also play an important role 
in this matter, opening up the field for further research. From the calculated results, we finally derived 
interventions for the upcoming course cycle aiming at turning the students’ initial extrinsic discussion 
motivation into a more intrinsic one. A comparison of the results after the next course cycle will then allow 
us to assess the effects of the implemented changes, which would not be possible without a critical thinking 
diagnosis approach like the one provided by Newman et al. (1995) and used in this paper. Also, the process 
of analyzing a forum in such detail helped us a lot to understand the effects of the chosen discussion task with 
its strength and weaknesses better. 
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ABSTRACT 

Traditional outdoor learning activities such as inquiry-based learning in nature science encounter many dilemmas. Due to 
prompt development of mobile computing and widespread of mobile devices, mobile learning becomes a big trend on 
education. The main purpose of this study is to develop a mobile -learning management system for overcoming the 
difficulties of outdoor learning activities. In addition, this study conducted a learning experiment on marine education in 
an elementary school for investigating its impact on the learners’ learning achievement and attitudes , and evaluating the 
suitability of this system. The results show that the mobile learning model with the system developed by this study has 
positive and significant effect on learners’ cognitive achievement. Nevertheless, the learners’ attitudes toward marine 
education is not enhanced after experiment. In addition, the result of system evaluation reveals that the experts show high 
appraisal toward this system including system usefulness, suitability and operational easiness. 


KEYWORDS 

Mobile Learning, 5E learning cycle, outdoor learning activity, nature science 


1. INTRODUCTION 

Inquiry is the process of humankind for seeking information and understanding, and it is a thinking approach 
as well (welsh Klopfer & Aikenhead, 1981). Inquiry-based learning is an open ended, student-center teaching 
strategy, emphasizing hands-on instructional context. In order to construct this kind of learning environment, 
there are many theoretical approaches proposed, including structured inquiry, guided inquiry, open inquiry, 
and learning cycle (Colburn, 2000). 

In inquiry-based learning (IBL for abbrev.), the students have to represent their problems, develop 
hypothesis, design experiment, collect and analyze data, propose evidence, and reflect their process. As a 
result, IBL can make learning more meaningful and enhance the effect of learning transfer (Brown & 
Campione, 1994; Collins, Brown & Holum, 1991; Linn & His, 2000; Slotta, 2002; White & Frederickson, 
1998; Lin, 2004). Nevertheless, IBL has not been widespread applied on educational practicum. Some 
literature revealed the possible reason of aforementioned phenomena including misunderstanding the 
meaning of inquiry, IBL regarded as suitable for high ability students merely, uneasy to manage learning 
activities, inappropriate teaching preparation (Colburn, 2000). In addition, Lai (2014) proposed some factors 
affecting the teachers’ rejecting the IBL, including delay the teaching schedule, bringing too much workload 
on teachers, the students’ time management problem, the teachers unfamiliar with IBL practical models, and 
lack of suitable digital management system. To reduce the aforementioned dilemma, some suitable learning 
models are required for practicum, and ICT-based learning management system is also necessary for 
delineating the teachers’ work load and providing real time supports for the students. 

Many IBL models offering concrete guidance can be suitably used for practical teaching and learning, 
such as Bishop and Bruce model (2002), Learning-for-use model (Edelson, 2001), and 5E learning cycle. The 
stages of Bishop and Bruce model consist of Ask, Investigate, Create, Discuss and Reflect. According to 
perspectives of constructivist, 5E learning cycle proposed by Biological Science Curriculum Improvement 
Study (BSCS) is regarded as an effective science pedagogy (Bybee & Landes, 1998; Bybee et al., 2006; Liu, 
Peng, Wu & Lin, 2009). 5E learning cycle is comprised of five stages including Engage, Explore, Explain, 
Elaborate and Evaluate. 
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2. SYSTEM FRAMEWORK AND FUNCTIONS 


In order to support outdoor science activities, this study developed a mobile learning management system 
(MLMS for abbrev.), and its framework is shown as Figure 1. MLMS consists of teacher-side learning 
management subsystem and learner-side learning support subsystem. 

Based on the theory of 5e learning cycle, the functions of learning support subsystem (Android -based 
APPs) for the students designed by Android java are depicted as follows: (1) Engage: To invoke the learners 
engage in the outdoor activities, the task-notify APP is designed, shown as Figure 1. (2)Explore: In this stage, 
the Kelly-grid APP can help the learners to observe the object specified by the teacher. Figure 2 presents a 
learning task of observing the crab of seashore specified in Kelly grid by the teacher. When the learners 
finish their task, the corresponding diagnostic message will be sent in mobile device for feedback or 
suggesting one furthermore observation if the student’s answer is incorrect. (3)Explain: In this stage, the 
teachers will provide the opportunities for explaining their learning, so the mobile -based worksheet App can 
facilitate the learner to collect and depict their findings by capturing image, annotating and writing text in 
their mobile devices, shown as Figure 3. When they encounter some problems, they can use QRCode for 
acquiring correct concepts or seeking solutions through Wikipedia and other search engines. (4)Elaborate: 
According to the cognitive elaboration theorem, the teachers can propose challenging and minds-on 
problems, and the learners are asked to solve it using elaborating concepts. In this study, the location -based 
knowledge construction App is offered when the learners are asked to finish a task, such as introducing the 
seashore spot of Taiwan, shown as Figure 3. (5)Evaluate: How to assess the learners’ performance is an 
important duty. In this study, the teachers can conduct peer assessment activities for evaluating the students’ 
works or ask the students to take a mobile-based test at the end of an outdoors activity. Figure 4 shows the 
peer assessment frame of mobile device and presents a mobile -bases test. Moreover, the profile of the 
learners will be recorded and upload into web server for further analysis. 

As for the teacher-side functions, the web-based learning management subsystem can be divided into the 
following modules: ( 1 )Kelly grid authoring: The teachers can edit Kelly grid for providing a series of 
observation task; (2) mobile test editing: The teachers can manage the test item bank and schedule a task of 
mobile test, as shown as Figure 5; (3) peer assessment management: The teachers can schedule a peer 
assessment task for some mobile worksheets; (4)task scheduling: All of task can be assigned and tuned by 
teachers. (5) viewing the learning status and outcomes: the teachers can monitor the learner’s task status, 
observation result in Kelly-grid, and score of test or peer assessment, as shown as Figure 6. 
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Figure 1. (A)partial system menu (B)Task-notify APP 


Figure 2. Kelly-grid and its feedback 
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(A) 


(B) 


Figure 3. (A) mobile worksheet (B) An example of location-based knowledge construction 

After finishing the development of this system and APP, this study employed 17 experts for evaluating 
the aforementioned system and APP, including 14 experienced teachers of elementary schools, 3 project 
managers, and 2 experienced programmers. The evaluation questionnaire adopted 5 -point Likert scale, 
ranking from strongly agree to strongly disagree. The facets of this instrument consist of system helpfulness, 
suitability and operational easiness. After analyzing the data collected from pilot study, the Cronbach’s alpha 
of each facet is over 0.7, revealing that this instrument is reliable. 
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Figure 4. (A) mobile -based test (B) peer assessment 
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Figure 5. Mobile test editing 
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Figure 6. Viewing the learners’ mobile worksheets 


3. OUTDOOR MOBILE LEARNING EXPERIMENT 

In order to investigate the learning effect of outdoor mobile learning, this study conducted a learning 
experiment by employing quasi experimental design. The subjects are 160 fifth graders from six classes in 
Taipei city, divided into experiment group (n=80) and control group (n=80). The experiment group accepted 
mobile learning approach, while the control group accepted traditional learning pedagogy. All of the subjects 
were asked to explore the plants and animals of specified seashore spot at northern Taiwan. The experiment 
group used Android Pads with aforementioned system and APPs for outdoor learning activities, on the 
contrary, the control group used traditional devices for recording their observation under directions of 
paper-based worksheets and learning materials. The educational target of outdoor learning of this activity is 
one of the elementary marine education. The subjects were asked to take cognitive tests of marine 
achievement and to fill out the self-reported inventory of learning attitudes toward marine education. The 
experimental design for outdoor activity in this learning experiment is shown as Table 1. 


Table 1. Experimental design for outdoor activity 


Pretest 

Treatment 

Posttest 

O1O2 

X 

O3O4 

o,o 2 


O3O4 


Oi : pretest of achievement test on marine education 
O2 : pretest of learning attitudes towards marine education 

X: experiment group accepts mobile learning model, and control group accepts traditional learning pedagogy 

0 3 : posttest of achievement test on marine education 

04 : posttest of learning attitudes towards marine education 


The inventory of learning attitudes consists of interest, usefulness, importance, and awareness toward 
marine education, adopting 5-point Likert scale. This instrument is validated by experts of marine educations. 
Its Cronbach’s alpha is .806 by analyzing the data collected from pilot test, indicating that it is highly 
reliable. The achievement test of marine education contains the concepts of plants and animals of seashores. 
After item analysis from pilot test, its difficulty index range from 0.4 to 0.7, and its discrimination index 
range from 0.4 to 0.8, depicting that they are all suitable for assessing the learners’ achievement based on 
viewpoint of Ebel and Frisbie (1991). 


4. RESEARCH RESULTS 

4.1 Results of Achievement test for Marine Education 

The descriptive statistical results of achievement test for marine education are shown as Table 2, and its 
results of independent t test are depicted as Table 3. As for pretest of achievement test, there is no significant 
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difference between experiment group (M=4.90, SD=2.949) and control group (M=4.31, SD=2.917), t= 1 .237, 
p>.05, indicating that those two groups possess the same prior knowledge on marine education before 
learning experiment. As for posttest of achievement test, there is significant difference between experiment 
group (M=9.61, SD=3.438) and control group (M=7.94, SD=4.018), t=2.727, p<.01, indicating that the 
experiment group divergently outperform on achievement test of marine education than the control group 
after learning experiment. In other words, the outdoor learning model with the aforementioned system and 
mobile devices can enhance the learning effect on marine education. The research results are same as those of 
Lai, Chou, Wu and Lai (2010) which employed RFID and PDA for conducting an experiment of learning 
aquatic plants in an elementary school. 


Table 2. Pretest and posttest scores of achievement test for marine education 


Test 

Group 

n 

Mean 

SD 


Experiment group 

73 

4.90 

2.949 

Pretest 

Control group 

77 

4.31 

2.917 


Experiment group 

72 

9.61 

3.438 

posttest 

Control group 

77 

7.94 

4.018 


Table 3. Results of independent t test on achievement test for marine education 



Levene’s Homogeneity test 

t 

df 

Sig. 


F 

Sig. 

pretest 

.046 

.831 

1.237 

148 

.218 

posttest 

2.877 

.092 

2.727** 

147 

.007 


w p<jrr 


4.2 Results of Learning Attitudes towards Marine Education 

The inventory of learning attitudes towards marine education is a kind of self-reported instrument, and it 
consists of four facets including interest, usefulness, importance, and awareness towards marine education. 
Its results of independent t test are depicted as Table 4. As for pretest of learning attitudes, there is no 
significant difference between experiment group (M=89.54, SD=13.311) and control group (M=91.1, 
SD=15.656), t=-.589, p>.05 , indicating that those attitudes of two groups are the same before learning 
experiment. As for posttest of learning attitudes, there is no significant difference between experiment group 
(M=89.95, SD=12.975) and control group (M=91.65, SD=14.436), t=-.706, p>.05. In other words, mobile 
learning model cannot promote the learners’ learning attitudes in a short-term leaning time. 


Table 4. Results of independent t test on learning attitudes towards marine education 


Learning attitudes 

Levene’s Homogeneity 
test 

t 

df 

Sig. 


F Sig. 




pretest 

.333 .565 

-.589 

122 

.557 

posttest 

1.191 .277 

-.706 

129 

.481 


4.3 The Results of System Evaluation 

The results of system evaluation are shown as Table 6, and average of each item range from 3.41 to 4.35. The 
average of this system evaluation is 4.016, revealing that the experts show high appraisal toward this system. 
Nevertheless, the system flow is required to revise in the future. 
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Table 5. The results of system evaluation 


Items 

This APP on mobile inquiry learning is helpful. 

This APP is useful for mobile inquiry learning in 
elementary school. 

This system recording the learners’ behaviors is 
helpful to capture learners’ status for the teachers 
The testing function of this APP is help for outdoor 
learning activities. 

The operational flow of this APP is smooth. 

This APP make outdoor learning more fun. 

This APP can attract the learners engage in learning. 
This mobile outdoor learning mobile is suitable. 

The multimedia materials through QRcode is helpful. 
Operating this system is easy. 

The web-based management of this system is easy to 
use. 

This system and APP can be used for different 
outdoor activities. 

This APP is suitable for indoor, campus-wide and 
outdoor learning activities. 

The functions of task-hinting is useful. 

The Kelly-grid and its real-time feedback is helpful 
for understanding their learning drawbacks. 

The mobile worksheet is suitable. 

The peer assessment of this system is helpful. 

The mobile worksheet integrating image and voice 
recording is interesting. 

If I am a teacher of nature science, I am willing to 
apply it in my teaching practicum. 

I am willing to recommend this system to other 
teachers. 


A(%) 

B(%) 

C(%) 

D(%) 

E(%) 

M 

SD 

23.5 

76.5 

0 

0 

0 

4.24 

.437 

35.3 

64.7 

0 

0 

0 

4.35 

.493 

11.8 

76.5 

11.8 

0 

0 

4.00 

.500 

5.9 

94.1 

0 

0 

0 

4.06 

.243 

11.8 

35.3 

41.2 

5.9 

5.9 

3.41 

1.00 

5.9 

88.2 

5.9 

0 

0 

4.00 

.354 

23.5 

70.6 

5.9 

0 

0 

4.18 

.529 

5.9 

76.5 

17.6 

0 

0 

3.88 

.485 

41.2 

41.2 

17.6 

0 

0 

4.24 

.752 

5.9 

58.8 

35.3 

0 

0 

3.71 

.588 

5.9 

58.8 

29.4 

5.9 

0 

3.65 

.702 

23.5 

76.5 

0 

0 

0 

4.24 

.437 

23.5 

58.8 

17.6 

0 

0 

4.06 

.659 

11.8 

82.4 

5.9 

0 

0 

4.06 

.429 

11.8 

52.9 

29.4 

5.9 

0 

3.71 

.772 

23.5 

76.5 

0 

0 

0 

4.24 

.437 

17.6 

76.5 

5.9 

0 

0 

4.12 

.485 

29.4 

70.6 

0 

0 

0 

4.29 

.470 

23.5 

64.7 

11.8 

0 

0 

4.12 

.600 

35.3 

47.1 

17.6 

0 

0 

4.18 

.728 


A: strongly agree. B: agree, C: medium, D: disagree, E: strongly disagree 


5. CONCLUSIONS AND FUTURE WORKS 

In order to offer real time support for outdoor learning activities, this study adopted mobile computing 
technology to develop a mobile learning management system and APP. Based on the theory of 5e learning 
cycle, this system provides different functions for each inquiry learning stage including Engage, Explore, 
Explain, Elaborate, and Evaluate stage, in outdoor inquiry-based learning. After system evaluation, the 
experts show high appraisal toward this system including system usefulness, suitability and operational 
easiness. Through short-term learning experiment on marine education, the outdoor mobile learning model 
can significantly improve the achievement of experiment group as compared to those of control group under 
traditional teaching pedagogy. Nevertheless, the learning attitude towards marine education is not promoted 
divergently after learning experiment. 

In the future, this study will launch a long term learning experiment for nature science, and apply it on 
different learning domain for investigating its effect on learners’ high order thinking skills including 
metacognition, communication skills and science process skills by using mobile learning model. 
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ABSTRACT 

Learning Style (LS) is an important parameter in the learning process. Therefore, learning styles should be considered in 
the design, development, and implementation of e-learning environments. Consequently, an important capability of an 
e-learning system could be the automatic determination of a student's learning style. In this paper, a set of features which 
are important in extracting the learning style automatically from students’ behavior has been determined. These features, 
which are recognized based on Myers-Briggs Type Indicator's (MBTI), play a key role in predicting learning styles in an 
online course. The features are determined and ranked using pattern recognition techniques, such as K-means clustering 
algorithm, to show which features can be better to separate learning style dimensions. The results show several features 
can be used to predict learning styles with high precision. 


KEYWORDS 

Learning style, e-leaming, MBTI, learner’s behavior 


1. INTRODUCTION 

Today, access to the web and the general use of computers have created several opportunities for e-learning 
systems, such as fully online and blended learning systems. E-learning environments, like all other tools, 
offer advantages such as access to different online resources, self-directed learning, and self-paced learning. 
Despite all the advantages, this kind of learning environment lacks the necessary attractiveness and the 
dynamic characteristic of the face-to-face learning setups. In other words, it is crucial to pay attention to the 
human characteristics and use them in the design and development of e-learning environments, aiming to 
make them more realistic and attractive [1]. This characteristic can give students the sensation that there is a 
human tutor involved in the process who follows their learning progress and cares about them [2], 
Consequently, matching an e-learning environment with individuals’ needs and personality characteristics 
helps them to learn more efficiently, than a system without adaptation to the learners’ characteristics. One of 
the most important characteristics in the learning process is the learning style which can be different between 
each two students. For example, while a group of students may like to listen and speak, the other prefer to 
analyze a text, or just listen to a visual media. When students cannot grasp a course’s materials, because of a 
mismatch with their learning styles, they lose their motivation for further learning [3]. This indicates that the 
better learning takes place when e-learning systems consider the learning style of the students. Thus, an 
automatic learning style recognition system significantly improves interactive training and learning systems. 

Several studies have been carried out in order to consider human characteristics such as learning styles in 
human computer interaction such as e-learning environments. In 2011, Gong and Wang [4] used support 
vector machine (SVM) in order to determine learning style in an e-learning environment. Haron suggested a 
learning system including a learning module which can be adapted to each learner and utilizes fuzzy logic 
and MBTI personality test [5], Abrahamian and his colleagues designed a user interface for learners based on 
their personality type using MBTI test [6]. They considered two dimensions of MBTI in their research. In [7], 
Bayesian networks is used in detecting the learning style of a student in a tutoring system. In [8], the 
students’ behavior in a web-based course were studied. Results indicated that students with different learning 
styles use different ways to learn and interact with the course. The authors also presented a framework for 
automatic detection of a learner's learning style based on the Felder-Silverman model. Fatahi and her 
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colleagues [9] designed and implemented a virtual tutor and virtual classmate agent that had personality and 
emotion features similar to a human being. They used the OCC model for emotion modeling and MBTI for 
personality modeling. The system was tested and the results showed that the learning quality and the 
learners’ satisfaction are improved by incorporating the human features in the interaction with the learners. 

As mentioned above, several studies have been carried out in order to consider human characteristics in 
e-learning environments. Although there have been few studies trying to extract a student's learning style and 
attempt to adapt the courses to it, none of these studies has taken into account the relationship between 
learning contents and learning styles. In this paper, we extracted a set of features which are important in 
determining learning styles from students’ behavior automatically based on the MBTI model. The proposed 
approach uses students’ behavioral features during the interaction with a learning content management 
system. We use clustering methods such as K-means to extract the best features that can distinguish learning 
style dimensions. In addition, we use MBTI personality type for detecting learning styles of learners whereas 
most of previous studies focused on Felder-Silverman learning styles. 


2. MATERIALS AND METHODS 

2.1 Learning Styles and MBTI 

According to the Keefe’s definition [10], “Learning style consists of cognitive, emotional, and physiological 
features which are used to recognize how the learner understands the concepts, interacts with the learning 
environment, and reacts to the environment”. There are many questionnaires categorizing each person in line 
with his/her learning styles [11]; such as Kolb's, Honey and Mumford's [12], GRSLSS's learning styles [12] 
and Felder-Silverman [13]. In comparison to the other questionnaires, the MBTI has been widely used and 
validated in the educational sciences [14]. It should be noted that MBTI can be considered as one of the most 
powerful tools to determine a learner's learning styles. 

MBTI is an assessment tool based on the Jung Personality Theory [15], Based on Jung’s personality 
theory, every person has instinctive priorities that specify their behavior in different conditions [16]. Jung’s 
type theory specifies three dimensions: Extraversion/Introversion (E/I); Sensing/Intuition (S/N); and 
Thinking/Feeling (T/F). Myers and Briggs added another dimension to Jung's typological model, i.e. 
Judging/Perceiving (J/P) [17]. These dimensions determined the learner’s learning style [17] [18]. Sixteen 
personality types result from mixing four two-dimensional functions. Individuals are categorized into these 
types after filling out the questionnaire [19]. For example, people in ESTJ group are all extrovert, sensing, 
thinking, and judging. 

2.2 Research Design 

In this paper, we focus on the learning style features that can be obtained automatically through a learner's 
behavior. In the following, a procedure for extracting the human behavior traits, based on a selected learning 
style model in an e-learning environment, is proposed here. 

//The algorithm for selecting the best online behavioral features, in an e-learning environment, to determine 
//learning Style (LS) 

//Output: The vector of best online behavioral features for predicting LS 
Step 1: Select a learning style (LS) model 

Step 2: Determine the behavioral features related to the selected LS 

Step 3: Determine the mapping between LS behavioral features and users’ behavior in the e-learning environment 
Step 4: Data collection 
For all students 

a. Determine their LS 

b. Determine features’ values based on logged behavior 
Step 5: Feature selection 

For each user’s behavioral features 

Find the relationship between each set of all dimensions of LS 
For each cluster of all LS dimensions 

If the relationship between LS dimensions and students’ behavioral feature is positive 
Add this feature to the vector of best features 
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In this paper, in the first step of the procedure, we chose MBTI which is an assessment tool related to 
Jung’s personality theory. According to the Center for Applications of Psychological Type [20], MBTI is the 
most commonly used personality inventory in history, with approximately 2,000,000 people have used MBTI 
for their personality detection every year. Moreover, the validity of MBTI theory has been widely shown [14] 
[21] [22]. The MBTI model is chosen and the general human behavior traits of this model related to the 
selected e-learning environment have been selected. In step 2, for each dimension, a few human behavior 
traits which are general and independent of the e-learning domain have been selected. These behaviors 
expressed in the most of studies about MBTI [18] . In step 3, we defined a mapping between LSs and a user’s 
behavior presented through the interaction logs based on psychological studies [18]. Our goal is to define 
features that can be extracted from e-learning logs which correspond to the LS behavioral features. For 
example, in the MBTI learning style model, psychologists suggest that extroverts are interested in group 
working [18]. Consequently, we expect them to have a large number of interactions in group-based and 
collaborative tasks such as forums, group projects, and team-based homework. In contrast, introverts prefer 
listening more than talking [18]; hence, it is anticipated that these learners prefer viewing forums and read 
posts more than writing in forums. The list of the features are presented in table 1 . 

Table 1. List of features extracted from Moodle 


Forums 

• The number of 
viewing 
discussions 

• The number of 
deleting 
discussions 

• The number of 
adding 
discussions 

• The number of 
viewing posts 

• The number of 
deleting posts 

• The number of 
adding posts 

• The number of 
updating posts 

• The frequency of 
viewing forums 
that includes 
students’ 
discussion 

• The frequency of 
viewing forums 
that includes TA 
news 


Homework 

Chat 

Folder 

The time of the 

• The number of messages 

• The number of 

first view of 

sent 

times viewing 

HWs files since 

• The chat view times 

slides of the book 

uploading 


• The number of 

The upload time 

Project 

times viewing the 

of aHW 

• The frequency of 

chapters of the 

The number of 

viewing discussion 

book folder 

viewing feedback 

about projects 

• The number of 

from a HW 

• The time of the first 

times viewing the 

The number of 

view of project files 

summery slides 

viewing HW file 

since uploading 

folder 

from the second 

• The number of times 

• The first time 

update to last 

viewing project files 

viewing the slides 

update 

Quiz 

of the book 

The number of 

• The number of viewing 

• The time of the 

viewing 

quiz page 

first book files’ 

homework file 

• The time of the first 

view since 

since the first to 

view of a quiz since 

uploading 

second update 

nrtl nuHt n ct 

• The time of the 

The number of 

UUlUclUlllL 

• The time spent on 

first book chapter 

viewing 


folder viewing 

homework file to 

answering quiz 

• The first time 

the first update 

• The number of reviews 

viewing the 

The number of 

in each question 

summery slides 

times viewing 

• The number of viewing 

folder 

homework 

quiz grade 


solutions 




Other 

• The notes view 
times 

• The blogs view 
times 

• The number of 
times using the 
search option 

• The changing the 
account password 
times 

• The number of 
times updating the 
profile 

• The number of 
times viewing the 
units information 
folder 

• The number of 
viewing the final 
exam sample files 

• The number of 
viewing the 
midterm exam 
sample files 

• The first time 
viewing the exam 
sample files 


In Step 4, in order to find the most important behavioral features to automatically determine the LS 
dimensions of a student, adequate data should be collected that can show a significant correlation between 
the behavioral features and the learning styles. Our goal is to select the best behavioral features that can 
determine the learning styles dimensions. Thus, in step 5, each feature is used to cluster the students into 
different groups. If the clustered groups highly correlate to a set of learning styles dimensions, then the 
feature is added to a feature vector suitable for determining the learning styles dimensions of a learner. 

To extract the important features based on the features explained in Table 1, we define features based on 
the context (Context-based Learning Activity Feature or CLAF for short). Since the context of learning 
activities could impact the interaction of a student with it, the CLAFs are determined considering the context 
of the activities. For instance, in our study, there were 9 homework in our course with different context 
resulting into nine CLAFs for a Learning Activity Feature. In our course, we came up with 112 CLAFs. 
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3. EXPERIMENTS 

This study is based on a blended learning environment for the “Introduction to computing systems and 
programming” (ICSP) course. The course is taught to the first-year students at the school of electrical and 
computer engineering at the University of Tehran in Iran. The course runs for 18 weeks, and is composed of 
two lectures and a laboratory part each week. The online part of the course includes general discussion 
forums, a place to receive and submit homework, download slides and notes, and check grades which are 
managed through Moodle Learning Content Management System. 

To evaluate our proposed algorithm, we have collected data from two hundred and twenty six students 
who participated in this study. At first, they registered for the course and filled the MBTI questionnaire to 
determine their learning styles, as our ground truth. However, only two hundred and three of these students 
filled out the questionnaire. The time student spent to complete the MBTI questionnaire was recorded. The 
data of students who spent less than three minutes or more than eighteen minutes on the MBTI questionnaire 
were discarded to avoid contaminating the data with unreliable data. Consequently, data from five students 
were discarded. Finally, data from one hundred ninety eight students were used in this study. Table 2 shows 
the learning styles' distribution for all dimensions. In addition. Figure 1 shows the distribution of personality 
types in each dimension of MBTI. 


Table 2. The distribution of the learning styles for the students 


Learning 

Styles 

Extroversion 

Introversion 

Sensing 

Intuition 

Thinking 

Feeling 

Judging 

Perceiving 

Number of 
Students 

69 

129 

108 

90 

149 

54 

106 

92 



Figure. 1 . Distribution of personality types at the school of ECE, University of Tehran, in each dimension of MBTI 

Although there are not equal number of people in each dimension, we selected equal number of samples 
to select best features in each dimensions, if we do not consider equal number of samples in each dimension, 
the clustering's results are biased toward the class with more samples. 

Total of 35,155 interaction records were collected of the students from the Moodle’s log file. Each record 
includes "time", "IP address", "action", "URL", "info", "username", "first name", "last name" and "email of 
the corresponding student". The "time" shows the duration of time students did an activity and the "info" 
includes an id uniquely assigned to the page accessed/used by the students. 


4. RESULTS 

As mentioned earlier, the data were collected, i.e. the learning styles of the students and their e-learning 
behavioral interactions with the system, for 198 records of students. As mentioned earlier, in our study, there 
were 9 homework in our course with different context resulting into nine CLAFs for a Learning Activity 
Feature. In our course, we came up with 112 CLAFs. 
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To determine the best features for determining the learning styles, we run many important clustering 
methods through Weka tools. K-means (k=2) was the suitable method for separating two MBTI dimensions. 
It seems clear that the k-means is one of the simplest algorithm which uses unsupervised learning method to 
solve known clustering issues. It is fast, robust and easier to understand. The k-means produce tighter clusters 
than other methods for example hierarchical clustering, especially if the clusters are globular. It gives best 
result when data set are distinct or well separated from each other. The disadvantage of the k-means method 
is predict k-value. In this research, this problem solved because there are two clusters in each dimension of 
personality type. 

Finally, k-means reports the better results rather than others. We run k-means method on all features. 
Tables 3, 4, 5, and 6 report which features have the high percentage of correctly clustered instances in each 
dimension in MBTI. 


Table 3. The list of context-based learning activity features to determine LSs in E/I dimension 


Feature 

Personality Dimension 

Extroversion 

Introversion 


Precision 

0.62 

0.63 

The response time of the Quizzes 

Recall 

0.66 

0.57 

Accuracy 

0.62 

0.62 


F-measure 

0.64 

0.60 


Table 4. The list of context-based learning activity features to determine LSs in S/N dimension 


Feature 

Personality Dimension 

Sensing 

Intuition 

The frequency of the folder view 
(the folders which includes the 
summery slide about each chapter 
of the course) 

Precision 

0.63 

0.94 

Recall 

0.63 

0.42 

Accuracy 

0.7 

0.70 

F-measure 

0.63 

0.58 

The frequency of the homework 
solution view (the homework 
which is about the assembly 
language) 

Precision 

0.70 

0.59 

Recall 

0.44 

0.81 

Accuracy 

0.62 

0.62 

F-measure 

0.54 

0.68 

The frequency of the homework 
solution view (the homework 
which is about the assembly 
language) 

Precision 

0.70 

0.59 

Recall 

0.44 

0.81 

Accuracy 

0.62 

0.62 

F-measure 

0.54 

0.68 

The number of times viewing HW j 
file (which is about flowchart) 

Precision 

0.70 

0.59 

Recall 

0.44 

0.81 

Accuracy 

0.62 

0.62 

F-measure 

0.54 

0.68 


Table 5. The list of context-based learning activity features to determine LSs in T/L dimension 


Feature 

Personality Dimension 

Thinking 

Feeling 

The frequency of viewing the 
forum (that includes the TAs 
news) 

Precision 

0.67 

0.73 

Recall 

0.78 

0.61 

Accuracy 

0.70 

0.70 

F-measure 

0.72 

0.67 

The number of times viewing the 
units information folder 

Precision 

1 

0.64 

Recall 

0.44 

1 

Accuracy 

0.72 

0.72 

F-measure 

0.61 

0.78 

The number of messages sent in 
the chat rooms 

Precision 

0.61 

0.87 

Recall 

0.94 

0.39 

Accuracy 

0.67 

0.67 

F-measure 

0.74 

0.54 
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Table 6. The list of context-based learning activity features to determine LSs in J/P dimension 


Feature 

Personality Dimension 

Perceiving 

Judging 

The time of the first view of the 
phase 3 file of the project since 
uploading 

Precision 

0.65 

0.63 

Recall 

0.62 

0.66 

Accuracy 

0.64 

0.64 

F-measure 

0.63 

0.64 


This paper aimed at determining the relationship between human behavioral features and learning styles 
dimensions in e-learning systems. Results in table 3, 4, 5 and 6 show there is a significant relationship 
between the human behaviors and the learning styles that can help to determine the learning style dimensions 
of a learner. 

Results in tables 3, 4, 5, and 6 indicate that there are features in each dimension of MBTI that are relevant 
to learners’ interaction with the e-learning system. Based on the fact that extroverted people tend to be fast in 
doing tasks, act quickly and sometimes without thinking, while the opposite is true for introverts. Our result 
confirms these behaviors because the response time of the quizzes is a feature that can cluster extroverts and 
introverts. Extrovert students have a high-speed in answering quizzes and do not waste time, but introvert 
students spend more time for thinking about the quizzes. 

Intuitive people attend more to the whole concept than to the details. In contrast, sensing people learn best 
from an ordered sequence of details, like to know the “right way” to solve problems, and wish to answer 
homework in details. Our result confirms such behavior. Frequency of folder viewing that includes summary 
slide about each chapter of the course is a suitable feature for clustering intuitive and sensing people. Results 
show intuitive people have more tendency to view summary slides than sensing people because they attend 
more to the whole than the details. Also, frequency of a detailed homework solution viewing, which was 
Homework 2 in our case, is an appropriate feature to separate sensing and intuitive students. This is because 
sensing students welcome details, especially difficult subjects such as assembly language which was the 
subject of the 2nd homework. The frequency of viewing homework solutions, especially related to algorithms 
and flowcharts, are well-suited features to separate S/N dimension. Sensing people learn best from an orderly 
sequence of details and love to know the “right way” to solve problems. 

Thinking people like specific and logical courses and objective subjects. Feeling people are personal, like 
warm people relationships, and they are more interested in people than things or ideas. Consequently the 
frequency of discussion viewing and frequency of posting to forums are the best features to separate feeling 
students from thinking ones. These features confirm that feeling people tend to interact and relate to other 
students through forums and discussion rooms. Table 5 shows that feeling students used chat rooms more 
than thinking people. Since thinking students prefer specific and detailed information, they used and 
downloaded materials from a special folder that included their program of study charts which explain the 
courses that the students should take during their four years of study. 

While the perceiving people have a flexible lifestyle and do not worry about deadlines, the judging people 
prefer a systematized life and regulated thoughts and ideas and care about activities which they can do on 
time. Our results show that the first time viewing files after uploading is a relevant feature that separates 
judging and perceiving students. Judging students are ordered, organized, and systematic, so they 
immediately view a file after uploading in the learning content management system. 

As the results show, there is one feature which is related to E/I and J/P dimensions while we initially 
estimated many behavioral features for these dimensions. This is due to the fact that many features, which are 
detectable in the face-to-face environment, are not detectable in e-learning environment. Consequently, we 
have a limited number of behavioral features in e-learning environments and the number of features vary 
based on the each dimension. 


5. CONCLUSION 

In this paper a set of features suitable for automatically determining the learning style dimensions of a learner 
is presented. These learners’ behavioral features, which have been extracted from the logged interaction of 
the learners with an e-learning system, have been determined using clustering approaches. We have used the 
MBTI model for identifying learning styles of students. The automatic recognition of users’ learning styles 
would be an important feature for intelligent tutoring systems’ designers to develop adaptive systems capable 
of adjusting to a user’s learning style. If these systems offer courses in harmony with learning styles of 
students, they have made a substantial progress in the learning process. A significant finding in this paper is 


71 



ISBN: 978-989-8533-40-1 ©2015 


finding a suitable feature vector to determine learning styles of students in automatic way. Furthermore, it has 
been shown that the context and content play an important role in the learning process based on learning 
styles. 

One limitation of the study is the number and diversity of the participants. Another limitation is the nature 
of blended learning structure of our e-learning system. Consequently, the students do not show all their 
behavior through online system and the off-line behaviors are not captured and used for learning style 
recognition. 

The findings in this study can be used as the basis for further research and improvements for providing 
advanced adaptability, especially in Learning Management Systems (LMS). Future work will deal with 
automatic student modeling process of learning styles and evaluating its effectiveness in improving the 
accuracy. Likewise, finding a sequence of human behavior will be of great value in predicting learning style. 
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ABSTRACT 

The present paper introduces some strategies used in teaching an Informatics course for non-IT first year university 
students. The strategies were developed during several years, using the main principles of flexible and blended learning. 
The methodology has been based on the results of experiments with students from social sciences, economics and 
technical faculties. Numerous tests were carried out to find out the level of prior knowledge and the preferred learning 
styles of each student. According to the results, students were divided into separate groups within an e-learning 
environment. 

The authors considered three main aspects: student’s background, the level of knowledge and the most suitable style of 
learning. The success of the experiment presented in this paper is demonstrated by comparing the academic results of the 
test group who was taught using the new strategy with the reference group, taught in a common way. 


KEYWORDS 

E-Learning, Flexible Learning, Felder-Silverman model. 


1. INTRODUCTION 

High level higher education is one of the most important aspects in the social development of a country. How 
can a learner get it? What can an educator do to provide it? First of all, both parties should be aware and 
using modern techniques that help raise learning effectiveness. 

At present there are so many opportunities to deliver information and knowledge, especially with the help 
of the World Wide Web. The Internet has quickly spread into all areas of human life around the world. In our 
country it is impossible to imagine an everyday activity without some e-components: e-government, 
e-school, e-health, etc. Furthermore, e-learning is also a rapidly developing possibility with various 
platforms. 

The main goal of e-education is to provide knowledge for each learner in the most suitable form. 
However, huge amounts of information from different sources do not guarantee perfect knowledge. 
Therefore, educational e-materials have to be carefully selected and structured to provide for the needs of the 
learners. It is a new paradigm of the present-day education. 

In modern times, all information and. as a result, all knowledge is rapidly changing, making it impossible 
for educators to predict exactly what the contemporary student will need to know tomorrow - on the next 
level of his study or in their future work. Instead, teachers have to be helping students to develop the 
necessary habits and skills which would enable students to learn whatever they might need in their future. 

At the moment. Informatics is a continuously and rapidly changing subject in the university curriculum. 
The main purpose of computing courses, therefore, should be to develop computational thinking and skills 
needed to use the standard PC equipment on an advanced level, as well as to help users understand and 
follow the latest trends in the field of IT. 

The contents of an Informatics course often seem to be inextricable for the first year non-IT students. 
There are several reasons for it: from low motivation to the absence of the real IT-projects during the first 
year at the university. Gradually, it has become obvious that IT-courses need to be innovated and the needs of 
the target audience have to be taken into account: we need more flexible learning possibilities and as much 
individualization in the teaching process as possible. 
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2. THE COURSE OUTLINE 

The Informatics course under discussion belongs to the curriculum of all specialities and it has been designed 
for the first year non-IT students. It lasts for two semesters and is taught in three languages. 

The main purpose of this course is to develop computational thinking by means of creating applications 
(using spreadsheets and some programming environments). The learning process starts with processing and 
analysing data in MS Excel: writing formulas, using built-in functions and drawing charts. The amount and 
topics of practical tasks and the theoretical material depend on the students’ faculty: social sciences, 
economics, chemistry, civil engineering or mechanics. 

In the second part of the course students learn the basics of modelling and programming in practice. 
Python and Visual Basic for Application (VBA) are the programming languages used. 

It should be mentioned that this part of the Informatics course used to be rather complicated for most of 
the students, especially students from the economics and social sciences departments. Difficulties at the start 
of programming tasks were solved by implementing graphical programming as an introduction to the course. 
After several years of experiments we chose the graphical programming environment Scratch as the main 
tool to introduce the main programming steps (MIT Media Lab, 2015). Scratch is a very intuitive language 
and greatly helps learners to take on board the main concepts and terms of modelling and programming such 
as the variable, subroutine, process, branching or cycle. 

During the course, we try to use and combine different styles of teaching and learning: the orthodox 
face-to-face classroom method, pair or group work and. especially, students’ independent learning in the 
e-environment. A learning environment such as Moodle (Moodle Trust, 2015) provides us with a great 
variety of different additional opportunities which enable making the learning process more flexible and 
more individual. A major way is the learning pace adjustment and variety in the number of learning 
assignments, which are based and matched with students’ level of knowledge and their learning styles. 


3. THE COURSE EVOLUTION 
3.1 The Background 

Unfortunately, teachers cannot provide all learners with one-to-one tutoring at all educational institutions. 
However, this fact has not affected the main aim of the educational process - to guarantee high-quality 
knowledge and modern skills. During the experiments with our course structure and content, we considered 
the differences in students’ characteristics, especially their background (the faculty and the level of prior 
knowledge) and their preferred learning styles. 

Since 2010, a group of lecturers started applying a new approach to the design of the computer science 
courses. Year by year this approach has become more flexible and adaptive to the nature of every student. 

At the beginning, we randomly divide all the students into equal reference and test groups. The division is 
not linked to the students’ specialization. The average number of members in each group is about 100-150; it 
varies depending on the general number of students. It should be noted that students are not aware of the 
research. The reference group is taught using the same course materials but these students are not supported 
with any additional systems. The students of the test group are directed in choosing their e-learning materials 
based on the data obtained through the tests in the e-environment. 

Our intention here is to compare the results of these two groups at the beginning and at the end of the 
course. 

At the beginning of the course the students are tested to find out their level of knowledge in the 
Informatics subject. Experience has shown that such testing is necessary for the development of the course 
content. The purpose is to keep track with new times and main trends, as the computer science is one of the 
fastest-developing sciences (College Board, 2015), (CSTA, 2015). 

The nature of the tests for both groups is similar and based on the concepts defined in the European 
Computer Driving License (ECDL) (ECDL Foundation, 2015). The assignments focus on some principles of 
the work with the PC, like creating text documents and presentations, handling information using the 
spreadsheets, and elementary programming knowledge. Tests include both practical and theoretical tasks. 
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The programming category of the questions was added to the test some years ago and is currently 
developing rapidly. We bear in mind that a new elective course “Basics of Application Development and 
Programming” was recently included in the secondary school curriculum in our country and assume that this 
is already showing the first results. 

The current situation of teaching computer sciences at schools in our country is quite discrepant. Some 
secondary schools do not have informatics lessons at all, in some schools it is taught only for two or three 
years, which is a very short period to prepare students for the university level. This drawback is associated 
with two main reasons. The first one being that there is no nationwide Informatics curriculum in our country. 
The second one is that Informatics subjects are not mandatory in our primary schools. A logical consequence 
of these reasons is the situation where each primary school teacher introduces learners to the material at his 
own discretion: certain pupils draw in Paint, others learn the computer hardware in theory, etc. 

In connection with this, the level of PC skills among non-IT learners falls every year and reduces to 
commonplace Facebook usage. The follow diagram (Fig. 1) shows the last year statistics about non-IT testing 
at the beginning of the Informatics course. The feasible maximum number of points is 100. As can be seen, 
the level of computer skills is quite low and steadily decreasing. 
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Figure 1. The averages of the beginning test 

We are also faced with another problem: school pupils who have obtained sufficient 

informatics -knowledge at their schools or additional courses often become IT students at university level. 
Unfortunately, they are not our audience. 

In the current work for innovating the Informatics course and individualization of the learning process the 
authors can name the main strategy - a maximally effective use of the e-environment with its possibilities. 
E-learning allows modern students (with a personal computer, connected to the internet) to attend the course 
anywhere and at any time. It is very important to get the material or submit the homework on time regardless 
of whether the student is or is not at the university. 

3.2 E-course 

The first phase, which we named “e-course”, includes the adaptation of educational materials for the 
e-environment in Moodle. 

Uploading a set of lecture materials and exercises into a learning environment does not ensure that 
students comprehend it and obtain necessary knowledge. Therefore, in order to make study materials suitable 
for an e-course all teaching materials (theory as well as materials for practice) were thoroughly revised. Our 
aim was to provide an effective delivery of the online content. To achieve this goal we aimed at working out 
a new pedagogical and didactic policy as well as strategies for the new e-course. 

Theoretical materials were innovated and supported with learning videos. It should be noted that in our 
course we use not only video lectures but also short screen -captures, which explain the most complicated 
tasks. Creating these videos, the authors adhered to the principles of Khan Academy (2015). Practical tasks 
of the course were reconsidered and supplemented with various group and pair work tasks and also self-tests. 
These innovations made the Informatics course more dynamic and attractive for our students. Both groups, 
the test and the reference group got access to this renewed course. 

At the same time, we have not abandoned the standard lessons in computer classes. They were held as 
usual but now we got many advantages. Due to the e-lectures and visual explanations in Moodle we had 
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more time for practical training in contact lessons. It is necessary to mention that students have no access to 
practical exercises unless they solve the tests which are based on the theoretical material of each topic. 
Thereby students come to the lessons already prepared for the practical tasks. Often they get a few small 
practical tasks in the e-environment and afterwards, in class, they use already ready solutions to solve the 
bigger tasks. 

It is generally known that effective computing is impossible without practice. During face-to-face lessons 
students work in pairs or groups and are given an opportunity to try the obtained knowledge in practice. 
Moreover, such kind of work develops teamwork skills, which is very important, especially for the first-year 
students during their first semester. 

In such case, the role of the classical educator is slightly different - the lecturer becomes more of a 
supporter in the students' team work of their learning assignments. During contact lessons learners have an 
opportunity to ask questions related to their homework and share their practical skills and experience with the 
rest of the group. Practical knowledge transferred in such a way is obtained much faster than in standard 
lectures. This fact was confirmed by the results of student tests and feedback. 

In their feedback students named another advantage of such practical lessons: they have an opportunity to 
get support or to ask something not only from their teacher but from other students, too. It should be 
mentioned that this form of support is equally important and useful for both sides: the one who gets it and. 
especially, for the one who gives it. To detect, explain and, afterwards, to correct a mistake in calculations or 
in the program code is a substantial practical skill in computer science subjects. 

In addition, in an e-environment students get their practical assignments in accordance with their 
specialities but still united under a common subject topic. For example, students from the economics 
department get more tasks related to table calculations; social sciences students implement the information 
filtration, statistics calculations and various requests. 

It should be noted that the course materials are organized sequentially and it is not possible to get a new 
portion of theoretical materials and practical tasks without solving the previous ones. Thanks to checking 
opportunities in the e-environment, like tests or self-tests, teachers do not have to spend time on routine 
inspection of the assignments at all. Using the automated checking tools gives students an opportunity to 
learn within their own pace. They do not have to wait for the feedback on the assignments from the lecturer 
and his/her manual permission to proceed onto the next level. E-checking systems do it faster and as many 
times as is needed. 

In addition to the aspects mentioned above we and our learners use forums very actively in our course. 
This way the students get an opportunity for online communication and fast online help. Our aim was to 
show that they can get support and advice any time, and can also share their ideas, problems and solutions. 
Through these forums we often get perfect brainchildren for group work and individual assignments. 

Another advantage that non-IT students have when they participate in the e-course is that their technical 
skills improve. 

Transition to the Moodle e-environment gave our educators an opportunity to follow students' progress 
and it became fairly easy to get the statistical data of different samples for analysis, development and 
improvement of the implemented learning methods. 

It makes no sense now to recount all the advantages that we achieved during the first stage of our 
modification of the course. Such benefits have already been systematized and described in detail in different 
sources (Broadbent, 2002). However, we would like to mention the first positive results of our work: 
students’ feedback and increase in academic achievements. The data was extracted from an e-environment, 
where students and teachers get information about the courses and curricula, students declare courses, keep 
results and give anonymous feedback on their educational process. The authors do not present specific data in 
numbers in this paper because the transition of the course into e-learning took place a long time ago. 

3.3 Prior Knowledge 

Starting from the second stage, which the authors entitle “prior knowledge”, experimental work takes place 
only with students from the test group. The students of the reference group are taught as usual. 

At the beginning of the course we start with dividing the students into three e-streams based on their 
readiness for Informatics subjects. This division was realized through an e-test and implemented in the 
Moodle environment. The students, however, were not aware of the experiment. 
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In the described division we proceeded from the level of students’ knowledge required to start the course. 
Those students whose e-test results are more than 85 points we named “experts”; “advanced users’” result is 
between 60 and 80 points, other students are called “beginners”. 

The named groups of students receive different amounts of practical and theoretical tasks in Moodle, with 
different levels of difficulty. To move to the next topic the mandatory set of exercises has to be solved. In the 
e-environment, the “beginners” and “advanced users’” have to solve their sets of tasks before they get access 
to “experts’” exercises - the main material of the course curriculum. “Advanced users” solve their tasks and 
can proceed to the main topics. 

These additional sets of tasks are catered for exactly what learners need to know for the current 
Informatics course. Students do not need to pay for any additional IT-courses and they get all the materials 
and assignments centrally, in one place - the Moodle e-course, in parallel with their main studies. 

To automate and speed up the checking of the increased number of tasks we have developed a special 
e-tests system. 

Thereby, we could increase the amount of practical assignments for students with different levels of 
readiness without increasing the subject hours and students’ load. This stage of the course innovation gave us 
an appropriate level of the students’ readiness for face-to-face lessons. 

The above mentioned method provided our Informatics teachers with actual and important information 
about what the learners knew before starting the course. Every year we get an overview of the current 
situation of Informatics subjects at secondary schools in our country. Moreover, according to the results, we 
are able to provide students with the necessary learning materials. 

3.4 Learning Style 

The third part of our innovation experiment with the course modification is the “learning style” phase and it 
was also realised in Moodle. This stage is a continuation of the test group’s students division into groups. 
There is a lot of research about the individualization of learning depending on students’ abilities (Kolb, 
1984), (Palmer, 2011), (George Lucas Educational Foundation, 2014). In our course Felder-Silverman model 
was picked out as the basis for the distribution (Felder & Spurlin, 2005), (Felder & Brent, 2005). In this stage 
we could maximize the use of the Moodle e-opportunities. 

Learning styles are characteristic cognitive, affective, and psychological behaviours that serve as 
relatively stable indicators of how learners perceive, interact with, and respond to the learning environment. 
Students learn best when instruction and learning context match their learning style. 

Depending on their learning style, Felder differentiates between the following groups of learners: 

- active and reflective 

- sensing and intuitive 

- visual and verbal 

- sequential and global. 

For better comprehension of the topic we give a brief description of the learners’ types. 

Active learners acquire new knowledge best by doing, discussing and explaining it to others in a group. 
At the same time reflective learners first think about it alone. 

Sensing learners like learning facts and solving problems by well-known methods. Intuitive learners 
prefer discovering new possibilities and relationships and they are more innovative. 

Visual learners remember pictures, diagrams, charts and videos best. Verbal learners prefer written and 
spoken explanations. 

Sequential learners like step by step studying, where each step follows logically from the previous one. 
Global learners prefer to get information by large portions and randomly. 

Through a test, we learned that the majority of our course participants in the test group were active and 
visual learners and they had very strong preferences for their learning process. These preferences were 
detected according to the Felder test, which was held at the beginning of the course. It should be noted that 
each year the number of active and. especially, visual students increases (Fig. 2). 
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Figure 2. The increase of the number of the visual and active learners 

Throughout the educational process, students were provided with necessary learning materials and 
activities in accordance with Felder’s instructions. 

For example, active learners automatically received more group work and opportunities to help others - 
they could check and correct other students' work and assignments in Moodle; they answered questions in 
the e-course forums and took the role of a tutor in face-to-face classes. It should be mentioned that they did it 
with pleasure. For visual learners, a great variety of visual representation of the educational materials (that 
was already mentioned above) was provided. Interests and preferences of the other types of learners were 
also taken into account. 


4. PROGRAMMING BASIS FOR VISUAL LEARNERS 


As mentioned above, the majority of the course participants are visual and active learners. For visual learners 
it is very important to see “how it works” and for active - “to try it out”. In the most complicated module of 
the Informatics course, programming, we found a solution - maximum visualization. 

Our introductory module, with Scratch as the main tool with its graphical elements, is already perfectly 
visualized. Scratch's dynamics and attractiveness helps visual students to understand the main ideas of the 
created applications. Working in this programming environment, active learners can manually modify some 
program elements and immediately see the result. Moreover, there are no syntax mistakes in Scratch that 
greatly facilitates the programming process. Therefore, introduction to programming in our course has 
already been visualized (Fig. 3). 

The next module, VBA or Python, is rather complicated for non-IT students. What can be visualized 
there? 

VBA already has a built-in visualising tool: students can follow the code execution using the Locals 
Window (Fig. 4). The named window automatically displays all the declared variables in the current 
procedure, their names, types and values in the real time. When the Locals Window is active and visible, it 
automatically updates during the time the program is running - visual students can follow the code and check 
each step and its result in their applications, especially if they run programs in step-by-step mode. 




Figure 3. Scratch program 


Figure 4. VBA visualizing tool - Locals Window 
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Besides the Locals Window it is possible to use the Immediate Window to reach the same goal: check and 
correct mistakes and understand the program structure. 

Unfortunately, Python does not have such an opportunity, but in spite of this, it still needs to be visualised 
for the beginners in programming. Therefore, we introduce the Online Python Tutor to our students. Using 
this internet application they can follow each step of their program code and check the variables’ values, 
types and the order of the operators during the execution. It has to be mentioned, though, that the online tool 
has some drawbacks, as it does not support the Python graphics, time functions and files processing. 
However, for the beginners in coding the application gives an understanding of the main principles of 
program construction and code execution (Fig. 5). There have been cases when students started to write the 
program not in Python environment but directly there, on the internet. 

Python 3.3 Frames Objects 

import random 
2 def randomsumma (many) : 
summa = 0 

^ 4 for i in range (many) : 

number = random. randint (—10, 10) 
summa += number 
print (summa) 
randomsumma ( 5 ) 

Edit code 

<= o= 

[ « First | [ < Back | Step 15 of 23 | Forward > | | Last» | 

line that has just executed 
^ next line to execute 

Figure 5. Python visualizing tool 

Using these online and built-in tools we could raise the interest to programming among the non-IT 
students in the second part of the Informatics course. 


Global frame 
random 
randomsumma 


module empty instance 


function 

randomsumma (many) 


randoms umma 

many 5 
summa -4 
i 2 

number -9 


5. COMPARING THE RESULTS 


Figure 6 shows the results of both groups' tests at the beginning and at the end of the course. The possible 
maximum number of points is 100. 



Figure 6. The testing results of the both groups 

As can be seen, the average test results of the reference group students are high enough - approximately 
76 points, but test group students’ average results are 16 points higher - about 92. The given outcomes 
confirm the topicality of our research and the suitability of the chosen method of the course innovation. 
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6. CONCLUSIONS 

During the process of upgrading the Informatics course, we were able to create a model of individualization 
of the educational process in an e-environment, which considers the level of students' prior knowledge and 
their preferences in the learning process (Fig. 7). 



Figure 7. The model of the course 

Applying this model, we try to find an individual approach to each student in our e-course and make it 
more flexible. 

The experiment results, presented in the article, show positive outcomes of the strategy and suggest that 
there are no unreachable aims in an educational process. The authors intend to continue with the created 
model and develop the Informatics e-course for non-IT students in the chosen direction, trying to maximally 
adapt it to students with different preferences. 


ACKNOWLEDGEMENT 

The authors thank anonymous EL2015 reviewers for their comments and suggestions. 


REFERENCES 

Broadbent, B., 2002. ABCs of e-Learning: Reaping the Benefits and Avoiding the Pitfalls. 1st ed. NY: John Wiley & 
Sons, Inc. 

College Board, 2015. AP Computer Science Principles. [Online] Available at: 
https://advancesinap.collegeboard.org/stem/computer-science-principles [Accessed 4 5 2015]. 

Computer Science Teachers Association, 2015. Computational Thinking Task Force. [Online] Available at: 
http://csta.acm.org/CiuTiculum/sub/CompThinking.html [Accessed 2 5 2015]. 

ECDL Foundation, 2015. ECDL Foundation. [Online] Available at: http://www.ecdl.com [Accessed 1 5 2015], 

Felder, R. & Brent, R., 2005. Understanding Students Differences. Journal of Engineering Education, 94( 1), pp. 57-72. 

Felder, R. M. & Spurlin, J. E., 2005. Applications, Reliability, and Validity of the Index of Learning Styles. Inti. Journal 
of Engineering Education, 21(1), pp. 103-112. 

George Lucas Educational Foundation, 2014. Project-Based Learning. [Online] Available at: 
http://www.edutopia.org/project-based-leaming [Accessed 24 11 2014]. 

Khan Academy, 2015. Khan Academy. [Online] Available at: https://www.khanacademy.org/ [Accessed 3 5 2015]. 

Kolb, D., 1984. Experiental Learning. Engle Cliffs: Prentice Hall. 

MIT Media Lab, 2015. Scratch - Imagine, Program, Share. [Online] Available at: http://scratch.mit.edu/ [Accessed 6 5 
2015]. 

Moodle Tmst, 2015. Open-source learning platform I Moodle.org. [Online] Available at: https://moodle.org/ [Accessed 6 
5 2015]. 

Palmer, S. R., 2011. The Lived Experience of Flexible Education - Theory, Policy and Practice. Journal of University 
Teaching & Learning Practice, 8(3). 


80 





International Conference e-Leaming 2015 


USABILITY OF A WEB -BASED SCHOOL EXPERIENCE 
SYSTEM: OPINIONS OF IT TEACHERS AND TEACHER 

CANDIDATES 

Zulfii Gen5 

Firat University 

Faculty of Education, Department of Computer Education and Instructional Technology, 23119, Elazig, Turkey 


ABSTRACT 

With advances in information and communication technologies, the classical nature of educational institutions has 
changed. One innovative effort within teacher training is the Web-Based School Experience System (WBSES) developed 
by the researcher. In this study, the usability of an existing WBSES is evaluated from both teachers' (n = 13) and students' 
(n = 96) perspectives. A non-experimental survey method was applied to collect participant opinions. A usability 
questionnaire was designed, and analysis yielded five significant factors for the system: usefulness, helpfulness, visual 
effectiveness, customizability, and smoothness. To obtain more in-depth information, participants were asked to delineate 
their responses to five open-ended questions. Generally, students and teachers liked the tool in terms of its usability, but 
they also provided recommendations for design issues. 
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1. INTRODUCTION 

Qualified human resources are imperative for modern societies to function and keep pace with the changing 
age. Teachers have a leading role in educating these human resources, socializing and preparing future 
generations by transferring culture and values to them (Ozden, 2005). The future of any country depends on 
the competence and success of its teachers (Ataiinal, 1997). Teachers shape human behaviors, making this 
profession both important and challenging. They are expected to be proficient in subject knowledge and 
content, teaching plans, general and specific teaching methods, and technology, requiring substantial 
education. Teacher training has been described as “a systematic program arranged for improving knowledge, 
skills and attitudes of teachers related to teaching and classroom activities” (Riza, 1996). In Turkey, 
important decisions have been made about teacher training, especially at the National Education Council, and 
the coordination of teacher training has been assigned to the education faculties of universities. 

Teacher training has two components in Turkey: theory and practice. School Experience and Teaching 
Practice are two essential courses that allow teacher candidates to develop knowledge, skills, and attitudes 
related to their profession. Fourth year education students gain their first teaching experience with these 
courses. The School Experience course takes place in the seventh semester and consists of one hour of theory 
and four hours of practice each week. In practice hours, teacher candidates observe at schools, learning about 
the environment and teaching. They also create lesson plans, notes, and other materials. 

In order for teacher training to be effective, the most important factor is cooperation between the faculty 
and the practice school. Effectively putting the theoretical and occupational knowledge of teacher candidates 
into practice is the fundamental objective of this course (Agaoglu and §im§ek, 2006). Therefore, cooperation 
between faculties, school administrators, course instructors, and teachers is quite important to its success. 
Known cooperation problems include an insufficient number of faculty members to mentor teacher 
candidates, weak communication between the faculty and practice school, a lack of meeting hours between 
teacher candidates and course instructors, lack of cooperation between department coordinators and practice 
school coordinators, and lack of program compliance (§ahin, 2004; Turgut, Yilmaz, & Firuzan, 2008). In a 
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study conducted in the Department of Computer Education and Instructional Technologies at Anadolu 
University, Dursun and Kuzu (2008) grouped such problems and their solutions under four sub-sections: 
consultation, communication, evaluation, and program. In order to eliminate these problems, close and 
effective cooperation between the faculty and practice schools should occur. A web-based learning model 
maybe a possible solution for improving cooperation in School Experience and Teaching Practice courses, as 
all processes could be easily conducted and monitored. 

A web-based learning environment supports and develops rich, flexible, and interactive activities by using 
the Internet, an intranet, or other technology components linked to a local network (§endag, 2008). Few 
studies have investigated the use of online or web-based systems in teacher training. In one experimental 
study, Liu (2005) concluded that a web-based cognitive apprenticeship model was more effective than a 
traditional teacher training course in terms of pre-service teacher performances and attitudes related to 
instructional planning. According to Barnett, Keating, Harwood, and Saam (2002), when pre-service teachers 
learn how to be good teachers in university-based courses, web-based conferencing provides further 
opportunities to interact and reflect with in-service teachers. In a study where a web-based learning model 
was used with teacher candidates from the English Language Teaching department of Anadolu University, 
Caner (2009) found that the model increased meeting hours between teacher candidates and instructors and 
formed a more productive learning environment by increasing peer feedback quality. The usability of 
web-based learning environments is very important to their acceptance by teachers and students. A central 
requirement is ease-of-use with minimal training. Usability refers to user interfaces and has five quality 
elements: leamability, efficiency, memorability, errors, and satisfaction (Nielsen, 2012). The ultimate goal of 
computer-based systems is tied closely to their usability and usefulness (Harms & Adams, 2008). 

The aim of the current study is to offer recommendations for developing better web-based learning 
environments in higher education institutions by analyzing the usability of the WBSES, which is already in 
use at Firat University. The study seeks to increase the effectiveness of the School Experience and Teaching 
Practice courses included in the curriculum of the education faculties by using a Web-Based School 
Experience System (WBSES) developed by the researcher. To this end. the system was tested with 96 fourth 
year students of the Computer and Instructional Technologies department of the Education Faculty of Firat 
University and 13 Information Technologies teachers in Elazig. The opinions of the teacher candidates and 
practice school teachers on the use of the WBSES were collected through a non -experimental survey method. 
The following research questions will be investigated in the current study: 

1. What are the opinions of the teacher candidates on the use of the Web-Based School Experience 
System for the Teaching Practice course? 

2. What are the opinions of the practice school teachers on the use of the Web-Based School 
Experience System for the Teaching Practice course? 

3. What are the satisfaction levels of the teacher candidates with the Teaching Practice course that 
applied the Web-Based School Experience System? 


2. WEB-BASED SCHOOL EXPERIENCE SYSTEM (WBSES) 

The general objective of the developed system is to provide an environment where all stages of the School 
Experience and Teaching Practice courses can be conducted through the Internet. The most important users 
of the system are the course instructors, practice school teachers, and teacher candidates. Access for these 
users is under the control of a program manager. This user assigns and connects schools, teacher candidates, 
school practice teachers, and course instructors. Course instructors conduct operations for their assigned 
teacher candidates, practice schools, and practice school teachers; practice school teachers conduct operations 
related to their teacher candidates. The program manager is the highest level user and has the following 
permissions: checking information related to faculty and department, matching teacher candidates to practice 
schools, assigning roles to users, authorizing user information, and creating questionnaires. The course 
instructor has the following permissions: updating personal information, arranging groups, accessing the 
videos and documents of teacher candidates, administering end-of-semester questionnaires, and monitoring 
attendance. Practice school teachers have similar permissions to the course instructor. Finally, teacher 
candidates use the program to share documents and videos and fill out forms and questionnaires. 
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3. METHOD 

The study aimed to determine the opinions of teacher candidates and practice school teachers about the use of 
the Web-Based School Experience System (WBSES) in the School Experience and Teaching Practice 
courses and was conducted via non-experimental survey. Qualitative and quantitative data collection were 
completed concurrently. Fraenkel and Wallen (2000) have identified three important problems related to such 
studies: (a) clarity of study questions, (b) honesty of self-reported answers, and (c) sufficient sample size. 
The first issue was resolved with factor analysis and a content validity check by field experts and linguists, 
and an assumption was made that participants would be honest in their answers, addressing the second 
problem. Elimination of the third problem was verified through statistical analysis of the data. 

3.1 Participants 

The participants included all teacher candidates attending the School Experience course offered by the 
education faculties of Turkey and their practice school teachers. According to Fraenkel and Wallen (2000), in 
situations where randomized or systematic sampling is not possible, researchers should use accessible 
sampling. Therefore, the researcher formed the sample with teachers and students in Elazig to which he had 
ready access. 


Table 1. Types of education and gender 


Gender 

Education Type 

Male 

Female 

Total 

First Program 

20 

28 

48 

Second Program 

12 

36 

48 

Total 

32 

64 

96 


As seen in Table 2, the majority of participants had their own Internet-connected computers. Most of the 
participants reported logging in to the system at least 15 times. 

Table 2. Computer ownership and use of WBSES 


WBSES Use 


Have computer 

5-10 

10-15 

15-20 

More than 20 

Total 

Yes-with Internet connection 

1 

13 

15 

32 

61 

Yes-without Internet connection 

1 

4 

17 

9 

31 

No computer 

0 

3 

1 

0 

4 

Total 

2 

20 

33 

41 

96 


The opinions of IT teachers in the practice schools (n =13) were also collected. Among these teachers, 
eight were female and five were male. Except one, they all had their own computers with Internet 
connections. The WBSES was accessed more than 20 times by five teachers, 15-20 times by two, 10-15 
times by five, and 5-10 times by one. 

3.2 Data Collection Tools and Analysis 

The first draft of the WBSES attitude scale was compiled based on studies investigated during the literature 
review. Since no appropriate scales were found, the researcher developed a new attitude scale for the 
WBSES. The item pool of the scale consisted of 57 items. After a language check, content validity was 
conducted based on the opinions of field experts. Participants were asked to complete a five-point Likert 
scale (“strongly agree” to “strongly disagree”). Four demographic questions were also added to the scale 
about participant education type, gender, computer ownership, and system-use frequency. In addition, the 
following five open-ended questions were asked to collect detailed data: 

1. What are your opinions ( visual design, navigation, etc.) of the WBSES? 

2. What are your three most liked features of the WBSES? 

3. What are your three most disliked features of the WBSES? 
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4. Did any functions cause trouble while using the WBSES? If so, which ones, and why did you 
have trouble? 

5. Which features do you believe aid ease of use of the WBSES ? Why? 

In order to determine satisfaction levels of teacher candidates, the 11 -item Web Based Teaching Practice 
Course Satisfaction Survey was adapted from Chejlyk (2006). 

Required permissions to implement the questionnaires were taken from the education faculty of Firat 
University and the Ministry of Education. All participants were informed about the developed system and 
process. At the end of the semester, the questionnaire was distributed on a voluntary basis to practice school 
teachers and teacher candidates using an online survey system (www.questionpro.com). Results were 
transferred to SPSS, and the frequency distribution of demographic information was calculated. Moreover, 
factor and validity analysis (Cronbach's alpha) were conducted on the 57-item questionnaire. After the 
modification of the scale, average and standard deviation calculations were completed. 

Tests related to the demographic features were also conducted. Education type (first or second education) 
and gender were tested by an independent sample t-test to reveal differences. In addition, gender was tested 
with an independent sample non-parametric Mann-Whitney U test, since no normal distribution existed due 
to the number of participants. An independent sample single factor analysis of variance (ANOVA), a 
parametric test, was used to determine possible differences between teacher candidates in terms of having a 
computer (with Internet connection, without Internet connection, or no computer); teachers were tested with 
an independent sample Kruskal-Wallis H test, a non-parametric test. To determine where possible differences 
occurred, necessary post-hoc tests were conducted. An ANOVA was also used to determine possible 
frequency differences between teacher candidates in terms of WBSES use (5-10 times, 10-15 times, 15-20 
times, more than 20 times); results were tested with an independent sample Kruskal-Wallis H test, a 
non-parametric test. To determine where possible differences occurred, necessary post-hoc tests were 
conducted. The five open-ended questions were investigated by qualitative techniques, and frequency 
distributions were calculated. 

4. FINDINGS 

4.1 Factor Analysis of the Attitude Scale of WBSES Use 

The student questionnaire had 57 five-point Likert questions (“strongly disagree” to “strongly agree”) 
answered via an online survey after finishing their exams. After the final data set, the questionnaire was 
checked by factor analysis. First, the Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy and 
Bartlett's test of sphericity were conducted. According to George and Mallery (2001), the KMO measure tests 
whether the distribution of values is sufficient for conducting factor analysis, and the coefficient should be 
greater than 0.60. Moreover, George and Mallery clarified that Bartlett's test of sphericity must be 
statistically significant as a measure of the multivariate normality of distribution. The KMO coefficient was 
found to be 0.87, and the approximate X2 (595, n = 95) was equal to 2525.350, p < .00. Since the tests were 
positive, factor analysis continued. 

The dimensionality of the 57 questions from the usability evaluation was analyzed using principal 
component exploratory factor analysis. Four criteria were used to determine the number of factors to rotate: 
the a priori hypothesis that the measure was unidimensional, the Cattell scree test, the variance explanation, 
and the interpretability of the factor solution. The scree pilot indicated that the initial hypothesis of 
unidimensionality was incorrect; there were five breaking points. After fifteen items were reduced by factor 
loadings and eigenvalues and when the interpretability criteria and total variance explained criteria were 
taken into consideration, five factors were rotated using a Varimax rotation. The rotated solution yielded five 
interpretable factors: usefulness, helpfulness, visual effectiveness, customizability, and smoothness. At the 
end of the factor analysis, the inter -reliability of each factor was calculated. Table 3 shows the coefficient 
alpha values for each factor, as well as all items in the questionnaire. 
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Table 3. Coefficient alpha values for each factor and all items 


Factor 

Number of items 

Coefficient alpha 

Usefulness 

12 

0.94 

Helpfulness 

6 

0.90 

Visual effectiveness 

4 

0.90 

Customizability 

6 

0.85 

Smoothness 

4 

0.78 

Total 

32 

0.96 


4.2 Student Usability Concerns about the WBSES 

fter factor analysis, a WBSES questionnaire was created with five dimensions; usefulness, helpfulness, visual 
effectiveness, customizability, and smoothness. Researchers checked each item to calculate minimum and 
maximum scores, means, and standard deviations. Afterwards, to determine whether the number of times 
using the tool made a significant difference on usability perceptions of students, a one-way ANOVA test was 
conducted on the data set. 

When the first dimension, usefulness, was statistically analyzed, mean scores varied from 3.57 to 4.08; in 
other words, students were satisfied with the usefulness of the system. The lowest score (M = 3.57) belonged 
to the item, “The video panel in WBSES (the video showing me while lecturing to be watched by my friends 
and teachers) made me active in the lesson.” For helpfulness, the mean scores ranged from 4.12 to 4.35, 
which confirmed that students were satisfied. Visual effectiveness scores ranged from 4.14 (the background 
color and text color were consistent) to 4.27 (overall visual attractiveness). Font, background color, and font 
color (M = 3.62) were acceptable to students, and the font color and background color were considered 
harmonious. Customizability mean scores ranged from 3.94 to 4.40, also indicating satisfaction. Students felt 
that their existing computer knowledge was sufficient to use the system (M = 4.40). Smoothness scores 
ranged from 3.40 to 4.36, and login and logout (M = 4.36) processes were easily managed. 

To gather more in-depth information, students were asked five open-ended questions. The first question 
involved overall ideas about the system (see Table 4). The majority of students (n = 59) described the system 
as visually good. Moreover, they felt the WBSES was useful (n = 18), easy to navigate (n = 12), and simple 
in design (n =11). One male student offered his impressions of the system: “I think it is a good system. 
Although we set up mail groups, which were unsuccessful, this system was successful. The system having a 
good visual design and easy navigation was especially effective in assignment uploading and filling out 
questionnaires.” The students also expressed complaints about the system (see Table 5). Negative comments 
were made about site navigation (n = 12) and the design of the system (n = 7). A female student observed, 
“Visual design of the system is good, but links not working when navigating through the system is a problem 
which can be solved easily. Moreover, the plug-ins needed to upload a document can be bothersome.” 
WBSES, though liked very much in terms of visual design by some, was not liked by others (n = 5). A male 
student expressed: “WBSES has a good design and appearance in general. However, I think the system is not 
useful because one time I forgot my password, but the feature for resetting the password was not working. I 
solved the password problem with the support of the system manager.” 

The second open-ended item asked for the three points that students liked most about the system (see 
Table 6). Answers included visual design, user-friendliness, the questionnaire, video and assignment upload 
capabilities, and online support. In the third question, students were asked what they did not like about the 
system (see Table 7). Six students expressed that there were no features that they disliked, while others 
complained about the uploading features, especially in terms of videos. Complaints were also made about 
accessing and staying connected to the system, as well as experiencing failure with different browsers. In 
addition, broken links were a problem. One male student explained, “The system does not allow uploading 
videos directly but through the YouTube links. The other thing I disliked is the requirement to log into the 
system frequently. The last problem with the system is that uploading long videos into the system was not 
possible.” In the fourth open-ended question, students identified processes they felt hard to manage (see 
Table 8). Many students (n = 23) experienced no difficulty with the tool. On the other hand, nearly half of the 
students had trouble uploading videos. In addition, students reported trouble with forgotten passwords, and 
three students had trouble accessing the system. The fifth question asked about processes students completed 
easily in the system (see Table 9). Generally, students mentioned navigation, uploading assignments, and 
filling out questionnaires. 
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Table 4. Positive opinions of teacher candidates about the Table 5. Negative opinions of teacher candidates about the 
use of WBSES use of WBSES 


Qualitative code Frequency Qualitative code Frequency 


A visually good system 

59 

Problematic navigation 

11 

Useful system 

18 

A need for improving the design 

7 

Easy navigation 

12 

Not useful 

5 

A simple design 

11 

Decreases in performance 

4 

Color harmony 

5 

Problematic color matching 

3 

Good content 

2 

Complex system 

3 

Assignment upload 

2 

Assignment upload 

3 

Site manager 

2 

Needs for installation of some plug-ins 

2 

Videos 

2 

Turkish character problem 

1 

Interesting 

1 



Easy to search 

1 



Questionnaires 

1 



Menu structure 

1 



Similar to Web 2.0 tools 

1 



Table 6. Most liked features of WBSES according to 

Table 7. Most disliked features of WBSES according 1 

teacher candidates 


teacher candidates 


Liked features 

Frequency 

Disliked features 

Frequency 

Visual design 

35 

Video upload 

30 

User friendliness 

25 

Logging out from the system 

21 

In-site navigation 

16 

Assignment upload 

17 

Questionnaire property 

16 

Access to the system 

14 

Video upload 

14 

System errors 

13 

Assignment upload 

14 

Broken links 

12 

Online support 

13 

Password problems 

10 

Simplicity 

12 

Visual design 

8 

Video watching 

10 

Filling out questionnaire 

7 

Colors 

10 

Color matching 

4 

Quick work 

7 

Font size 

3 

Attendance 

6 

Problems with different browsers 

2 

Support for teaching practice 

6 

Photograph upload 

1 

Easy access to the system 

5 

Lecture content 

1 

Peer evaluation 

2 

Lacking of chat feature 

1 

Effective searching 

2 



Table 8. Problems experienced with the WBSES by 

Table 9. Ease of using WBSES features 

teacher candidates 






Ease of use 

Frequency 

Problems experienced 

Frequency 

Navigation 

17 

Video upload 

41 

Assignment upload 

13 

Assignment upload 

21 

Questionnaire use 

12 

Getting user password 

9 

Video upload 

8 

Filling out questionnaire 

7 

Membership system 

5 

Access to the system 

3 

Visual design 

5 

Navigation through the links 

2 

Fast working 

4 

Working with different browsers 

1 

Contact with the system manager 

3 



Attendance check 

2 



Access to the system 

2 



Video watching 

1 



Simplicity of the interface 

1 
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4.3 Practice School Teachers' Usability Concerns about the WBSES 

The teacher questionnaire included 32 questions, but because only 13 teachers responded, it could not be 
factor analyzed. On the other hand, since it asked the same questions (except student-oriented questions), it 
was still used. The 13 teachers who responded used the system approximately one hundred times. In general, 
all instructors had positive ideas about the login and logout processes and noted that the tool was easy to 
learn. They were moderately satisfied with the visual screen design. Mostly, they reported no problems using 
the system. 

The teacher questionnaire also included five open-ended questions: 

1 . What is your overall opinion about the tool? 

2. List the three features of the tool that you liked most. 

3. List the three features of the tool that you liked least. 

4. When you used the tool, were any processes difficult? How so? 

5. When you used the tool, were any processes completed easily? How so? 

Five teachers expressed that the system was useful, and six teachers considered the system good in terms 
of visual design. Moreover, six teachers expressed that the system was successful in terms of color harmony. 
Four teachers felt the system had easy navigational features. Only one teacher had negative opinions about 
the design. One female teacher expressed, “The design and color matching in the system are really nice. The 
web-site was useful and I can do all the operations easily.” The most liked feature by the teachers was the 
attendance panel. Three teachers liked the video features best, including uploading capabilities and watching 
student contributions. One teacher expressed that the system was very useful in that it allowed students to 
realize their mistakes by watching videos. The other most liked feature was the ability to create 
questionnaires (n = 7). Another teacher liked the visual design, response time, web mapping, and access to 
the faculty instructor in the system. 

Among 13 practice school teachers, 9 expressed that there were no features that they disliked. One 
teacher did mention early problems that were resolved over time. One teacher expressed a complaint about 
broken links, while another did not like the content arrangement. In the fourth open-ended question, nine 
teachers noted that they experienced no problems. One teacher said that after accessing the system, he did not 
see the expected screen. Another teacher described problems with the attendance feature, since there was no 
grouping according to date. In addition, one teacher complained about a failure sending messages. Two 
teachers did not answer the last question, while five said that all features of the system were user friendly. 
One teacher said that he liked working without paper, and another expressed that he easily completed all 
operations without the need to refresh the page. One teacher commented, “In general, there are features that 
make the system user-friendly. Especially, I found the access panel and design of the system successful. In 
addition, navigation between the pages was very fast and stable.” 

4.4 Teacher Candidate Satisfaction Levels with the WBSES 

The averages and standard deviations of answers given by teacher candidates about the WBSES course 
conducted through the web are shown in Table 10. The satisfaction levels are grouped under four headings: 
overall satisfaction, contribution to professional development satisfaction, feedback satisfaction, and online 
communication and interaction satisfaction. 

Based on these results, teacher candidates were generally satisfied with the course. The responses of 
teacher candidates to item 6 showed that 75% were satisfied and 14.29% were strongly satisfied with the 
web-based school experience course, and 85.72% of participants strongly agreed that they would recommend 
the course to others. Analysis revealed that 82.14% of participants considered this course as effective as 
face-to-face courses. According to results, teacher candidates were satisfied with the contribution of this 
course to their professional development. They also expressed that course documents, lesson plans, and 
lesson practice videos facilitated their learning and that this web-based course was as useful as a face-to-face 
course. The teacher candidates’ responses to the satisfaction survey showed that they were satisfied with the 
timely feedback from their teachers and the system manager. In terms of satisfaction with online 
communication and interaction, participants believed that this course provided a sense of community among 
the students. 
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Table 10. Satisfaction levels of teacher candidates with the WBSES 


j Question Item 

N 

Average 

S.D. | 

| Overall Satisfaction I 

6. 1 am very satisfied with this web based school experience course. 

96 

4.04 

0.50 

9. 1 would recommend this course to others. 

96 

3.79 

1.13 

11.1 feel the web-based school experience course is as effective as face-to-face courses. 

96 

3.68 

0.81 

1 Contribution to Professional Development Satisfaction 1 

1. The course documents, lesson plans, and lesson-practice video files used in this class 
facilitated my learning. 

96 

3.71 

1.01 

3. The materials that were linked to this course facilitated my learning. 

96 

3.82 

0.54 

7. 1 feel this web-based school experience course improved my teaching skills. 

96 

3.86 

0.50 

10. 1 learned at least as much as I would have in a face-to-face course. 

96 

3.86 

0.80 

| Feedback Satisfaction | 

| 2. 1 received timely feedback from my teacher and system administrator. 

96 

3.75 

0.84 | 

| Online Communication and Interaction Satisfaction j 

4. This course created a sense of community among students. 

96 

3.79 

0.68 

5. In this class, I was able to share my viewpoint with other students. 

96 

3.86 

0.75 

8. This web-based teaching experience course encouraged students to discuss ideas and 
concepts with other students. 

96 

3.68 

0.86 


5. DISCUSSION AND SUGGESTIONS 

Literature related to teaching practices reveals that teacher candidates often encounter problems with 
sufficient feedback and meetings with course instructors. Lack of cooperation between the faculty and 
practice school and inadequate duration of implementation are other known problems. A solution should be 
found to eliminate these problems in order for teaching practice courses to be effective. To this end, a 
web-based system to be used in teaching practice and school experience courses was developed in the first 
stage of this study. In the second stage, usability analysis of the system was conducted by collecting the 
opinions of practice school teachers and teacher candidates. Moreover, satisfaction levels of teacher 
candidates related to web-based learning were investigated in order to determine whether the system met 
learning needs. The study primarily focused on the design, development, usability, and formative evaluation 
of a teaching practice course conducted through a web-based system. 

The majority of the users defined the system as having high usability. Moreover, the teacher candidates 
expressed that system managers having an active role in the system increased usability. They expressed that 
the system contributed to the course and met their learning needs. The opinions of the teacher candidates 
towards the WBSES was positive. They believed that having taken the course with this model contributed to 
their professional growth. In summary, the web-based teaching training model increased course 
effectiveness, enhancing the performance and experience of the teacher candidates. 

The web-based system in the current study was tested with students from the Computer and Instruction 
Technologies department and IT teachers with high computer knowledge. Therefore, this study should be 
repeated with students from other departments in order to investigate the attitudes of teachers and teacher 
candidates with less computer experience. 
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ABSTRACT 

Hearing impairment may constitute a barrier for accessing to information and communication in public places. Since the 
oral communication forms the basis of the learning process, this problem becomes of particular relevance at schools and 
universities. To cope with this situation is not enough to provide a textual translation for people with hearing disabilities, 
society via educational authorities must facilitate alternatives that improve access to information and education to this 
collective. According to this reality, the possibility of having an alternative tool of communication based in the Spanish 
Sign Language (SSL) emerges as a contribution to help overcoming the communication obstacles that the students with 
this difficulty usually find. The Spanish law 27/2007 recognizes and regulates that the sign language has to be used as the 
means of support for oral communication for deaf, hearing disability and deaf-blind people. It also determines the rights 
of the people to decide loosely how to communicate, to not to be discriminated because of it and lays different 
application areas. According to this law, the design of a communication system based on SSL is proposed, with the aim 
of contributing to the Universal Accessibility as indicated by the International Convention on the Rights of the Persons 
with Disability. 


KEYWORDS 
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1. INTRODUCTION 

Inside the group of hearing impaired people, many of them, as first way of communication and in some cases 
as the only way, use the sign language. In this sense, Fuertes and others indicate that the access to higher 
education is “one of the most complex problems that the groups of hearing impaired people have to face” [1]. 
The same authors show that to solve this problem, it is required “the combination between the Spanish 
language and the SSL”. 

Obviously, the access to higher education is an integration problem and not about intelligence for the 
hearing impaired. In their path arriving to university, many students that have had this disability since they 
were born have been using the SSL as their habitual language of communication. However, when applying to 
the higher education, they find a model of communication based in the oral and written language which is 
used by the people without disability and that it is not adapted to their necessities and knowledge. 

In the particular case of people that do not feel comfortable with this disability, it is necessary to put at 
their reach all the available resources to access information "to the knowledge and independent learning, 
without which the technologies of information and communication, which may be an opportunity, become a 
new barrier" [2]. An example of this could be in the widely attended public places (airports, subway 
stations...) where the messages of general interest for the users are not received by everybody easily. For 
some of the people who suffer hearing disability it is difficult and in many cases is impossible to receive 
those messages. Things like this make us think in the new technologies, with international design patterns, 
favoring a true framework of inclusion in the society [3]. In this sense, over the last few years many 
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improvements have been created to facilitate the communication and the access to information for the people 
who suffer hearing disability. Between these facilities, it stands the installation of magnetic loops in public 
places to provide audible information that makes it clearer for the people who use hearing aid or cochlear 
implant. However, these interventions do not completely solve the communication problems which people 
who only use the sign language as the only way of communication usually encounter. 

As it is known, “there is not a universal sign language” [4]. Every country has its own sign language. 
Even inside every Spanish community there are variants of this language. Due to this situation, for the 
creation of the application which is raised, the Spanish language was selected as the official language in the 
Law 27/2007, in the 23 rd October, which states that the Spanish sign languages were recognized and the 
means of support for oral communication for deaf, hearing impaired and deaf-blind people were regulated 
[5]. Inside Spain, there are some variations between the sign languages that are used in the different parts of 
the country, exceptionally Cataluna, where those are more significant. 

Nowadays, the majority of universities use virtual learning platforms to support classroom teaching and. 
in some cases, “e-learning” for distance learning. These systems can contribute as a support tool to the group 
of hearing impaired, independently of the education levels they are in. The objective is to obtain the 
information that is given in class and to deepen in it more carefully. However, this solution does not 
completely solve the problem neither in the classroom nor in the learning platform, as both of these options 
do not give support tools based in the sign language [1]. 


2. PURPOSE AND OBJECTIVES 

The purpose of the study consists in the integration of the students with a significant hearing impairing and 
those who use the SSL as their way of communication through the projection of subtitled videos, where an 
avatar presents the translation of the oral expositions that the teacher have created in sign language. 

The proposal raised here consists in an inclusive intervention which can be implemented in any learning 
level, from primary school to higher education. In this case, it focuses on a university for the higher 
knowledge and its performance. With this proposal, it is almost simultaneous that the oral expositions which 
the teachers imparts in the classroom are shown with videos in sign language, which expose the same 
contents and even introduces the subtitles with the corresponding text, but translated to SSL to cover a 
broader spectrum of communication. Also, these videos will be in the virtual campus, with the rest of the 
documentation of the subject. With this action we pretend to facilitate the access to the information to the 
students with this disability, contributing to the independence and integration of these people in the society, 
as well as the improvement of the quality of life and the fulfillment of the “Universal Accessibility of the 
people with disability”. 

It is a pilot project due to perform in a concrete subject. To start up this intervention, we will choose a 
place where there are students who have this disability and demand support through the SSL. The design 
requires a coordinate action between the teachers and the Office of Learning support (GAP) of the ULPGC, 
who will give the necessary support for the preparation of the exposed texts through the subtitles in the 
videos and will be responsible for the translation to SSL. 

The recording of the translations will be made separately for each class session. Then a multimedia 
installation will be performed by inserting subtitles and converting the real image into an avatar. The use of 
an animated character in two-dimensions (2D) is justified with the goal of dispensing real people’s faces and 
avoids the monotony in audiovisual generated animations. In this sense, we have taken into account the 
positive valuations that the members of the Association of People with Hearing Disability in Gran Canaria 
did, who were asked about the project. 84% of the people asked, did agree with the use of an avatar in 2D 
instead of an individual. 

In turn, for each class, small videos explaining each section will be performed. This way, we will have 
multiple videos of short duration for each topic of the class, which you can access through a graphical 
environment that allows the organization per unit and paragraph referenced within each topic. The 
performance of the graphical environment will aid the students to locate and access in a structured and fast 
way the concept he wants to play. Likewise, the use of the short duration videos is aimed at avoiding the 
distraction during the view or an excessive tiredness at the time of its viewing. 
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Finally, the inclusion of subtitles comes motivated by the possibility of offering all the students the use 
this multimedia material to understand the explanation of the contents which are being treated, so it is not 
only a useful tool for the hearing impaired, but also for the rest of students. 


3. METHODOLOGY 

The performance of this project is structured in different stages, which keep a direct and sequential 
interrelationship between them: center and subject selection, faculty coordination with the GAP, elaboration 
of a screenplay with the preparation of the class, translation to the sign language, design of the character and 
the conversion of the videos to animation in 2D and the performance of an interactive graphical environment 
for the call and presentation of animations. 

Figure 1 shows the sequencing phases of the proposed work for this project’s development. 



Figure 1. Phases for the elaboration of the project 

3.1 Center and Subject Selection 

The first step of the implementation of the project will be the selection a teaching center in which we can find 
students with hearing disability and they request support through translation to SSL. 

Once the center is chosen, the next step will be to select the subject which is going to be intervened. A 
basic subject is proposed, where these students highly and where the first priority is the science. The reason 
of choosing a basic subject is because these subjects are imparted in several degrees of the same branch, with 
identical or very similar contents, so it would benefit a higher number of students. Meanwhile the 
predilection for the science area is justified by having a higher content of mathematic issues and fewer texts 
than a subject of another area, which reduces and simplifies the translation to SSL and the elaboration of a 
final multimedia product. Based on these arguments, the “Calculus” subject is chosen for the implementation 
of this pilot project, meeting the premises established. 

3.2 Coordination of the Teaching Staff with the GAP 

This is a multidisciplinary project, covering the teaching staff, the GAP of the university and the responsible 
of the audiovisuals performance. Due to this, it requires coordination between the different parts and a clear 
and concise planning, with the statement of each person’s responsability and a performance schedule which 
allows achieving the final product, with the programmed quality and in the stipulated dates. 

3.3 Elaboration of a Screenplay with the Preparation of the Class 

This task will take on a previous work of selection and preparation of the texts for its posterior translation to 
the sign language. Since in the chosen subject you have to work on a board and there are many expressions 
which are repeated, a previous work should be made, where those common phrases are collected. This will 
suppose a significant saving of the subsequent tasks. The next task will consist in the performance of a 
screenplay with recorded texts. The recordings will be made by the faculty, experts of the subject, who will 
choose texts which will be translated to SSL and will be included as subtitles in the final video. Therefore, in 
this phase we will analyze and synthesize the net language used as pilot project. 
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3.4 Translation to Sign Language 

After the elaboration of the screenplay of each class and the selection of the texts, the next step in the 
execution of this multimedia project will consist in the translation of these texts to SSL. To performance this 
step, is necessary to count with an interpreter of SSL. This task will be entrusted to the GAP, who trough his 
Translation Cabinet is responsible of the translation of the selected material. To avoid the confusions in these 
phases of the project, we must record each class in different and independent videos, classifying them with 
the numeration and identification of each topic. 

3.5 Character Design and Conversion of 2D Animation Videos 

Currently there are many programs for creating vector animations easily, offering all kinds of tools which 
range from the most basic to others that require more skills and licensing fees. There are others which are 
completely free, although these generally do not have the same benefits. Depending on the level of animation 
it is wanted to achieve, it should be chosen the one that best suits the requirements of the final product. At 
this phase, an analysis of the main tools available today was made in the field of computer animation, both 
professional and amateur. The most common vector animation programs can be differenced in two types: free 
open source, like Ajax Animator, KToon, Pencil or Synfig, and the ones which require a payment, such as 
Adobe Flash, Adobe AfterEffects, Anime Studio or Toon Boom, among others. In this sense, and with the 
goal of facilitating the choice, it is proposed the use of a specific software for animation, known as Adobe 
Flash CS5, which is an application shaped studio working on "frames” and it is intended to the production 
and delivery of interactive content. It has been used in similar projects, obtaining a high level of satisfaction. 

It uses vector graphics and raster images (bitmap), sound, programs code streaming video and two-way 
audio. In addition, this program incorporates an animation technique called rotoscoping. Its advantage is that 
its use is that it can replace the faces of people, who can be more or less attractive, for an animation 
character. 

The use of the rotoscoping in this project would be very appropriate for obtaining vector animations. With 
this animation and the incorporation of some of the elements in the scene as a stage, wallpaper, etc. the final 
videos in SSL would be obtained, in which the interpreter becomes an animated character. 

In figure 2, a rotoscoping process is shown, where the conversion of video frame is performed to a 2D 
animation. 



Figure 2. Changes in a single frame, during the animation process. 
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The animation consists of image succession, in order that when they are showed with a determinate 
speed, they produce the feeling of movement. Adobe Flash has some amenities when animating elements 
because of it interpolated animation function. The produced changes on the objects are performance easily 
and efficiently by setting key frames. This marks the initial and final points of the animation. Adobe Flash 
automatically generates the content of the intermediate frames modifying the necessary parameters, like the 
position, the size, the rotation or even the form of an object [7]. 

In the development of the videos, formal aspects must be taken care of, such as the scene background 
colors, the character, the speed of the gestures, the inclusion of subtitles or symbology, with the goal of 
capturing the users’ attention. 

3.6 The Performance of an Interactive Graphic Environment for the Call and 
the Presentation of the Animations 


As it has been commented before, it is pretended to obtain short duration videos in SSL, thus leading to a 
large number of them. Due to this and with the goal of facilitating their identification and call in a quick and 
ordered way, it is proposed the creation of a multimedia app that facilitates the access to the users and the 
selection of the videos. It is about the design of a simple setting, interactive app which allows the interaction 
of the users with the shown elements in the screen, such as folders with the number and name of the topic. 
Inside each folder will be placed call buttons to videos, created for that particular subject. By selecting each 
of these buttons, which will have the name of the concept it represents, the corresponding video will start. In 
turn, in the videos the corresponding start, forward, pause, and final buttons should be placed. These orders 
will be made through a peripheral that controls these interfaces, to obtain the requested information. For the 
development and programming of these apps, it is proposed the software “ActionScript 3.0”, which is the 
native programming language of Flash. 

Without prejudice to what was mentioned earlier, it would be convenient, once the first version of the 
graphic environment is performed, to solicit opinions and suggestions from experts in SSL communication, 
like the GAP of the university. In this way, it would be obtained a customized product directed to the real 
needs and demands of the targeted population and its chances of success will be greater. 

A technical detail to be considered for the correct performance of this app is that the videos in SSL should 
be stored in the same medium in which the multimedia application is, to obtain a greater speed in the call and 
reproduction of the videos. 

In the figures 3 and 4 it is shown a first approximation of the proposal made for this project. 



Figure 3. Principal page of the graphic environment. 
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Figure 4. Screen for the selection of the videos. 


4. CONCLUSIONS 

The performance and implementation of this project would make the access to the information that is 
provided in the class to students that suffer hearing impairment easier. Furthermore, the same compliance 
would be given to the law on disability, in relation to "the rights of individuals to decide freely the media and 
not to be discriminated for it." 

On the other hand, it would contribute to a didactic material in SSL available for the students which 
demand it in this university, and which can be shared with other Spanish university, provided that subjects 
chosen for translation are basics subjects, because the system of higher education is included in the European 
Fhgher Education Area (EFfEA), which uniformity is required in basic subjects. Its use in an interuniversity 
level, would suppose that this project has impact at national level, thereby saving cost and provide a fast 
payback. 

Once this project is implemented and evaluated, it could be made extensible to the rest of basics subjects 
and even extended to other degrees, especially in those subjects in which the agenda is not susceptible to 
significant changes. 

The teaching material that is generated will be placed in the virtual campus, available to all students, so 
they can access it in any moment, either during the course of the class or after the them. Also, it allows the 
option of using multiple multimedia devices for downloading and viewing: desktop computers or laptops, 
tablets, smartphones, etc. 

This project can be extensible to other levels of education which require teaching in SSL. It can be useful 
for partnerships with People with Hearing Disability, for the performance of on-line courses in SSL about 
issues demanded by its partners. 

In the search of information for the development of this article, a large quantity of instructional videos 
about teaching support were found, but in any case these were teaching SSL. 
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ABSTRACT 

The use of information and communication technologies (ICT) and therefore e-learning is becoming an ever more 
frequently used teaching and learning technique at all levels of education. In higher education, it completely or partially 
substitutes the classical teaching methods. It provides richer resources than the traditional classroom and overcomes its 
limitations (time and space). In some cases, it has proved to be very effective. Some authors, however, also point out that 
e-learning requires highly self-regulated and independent students or their e-learning performance may be low. The 
purpose of the paper is to analyse how introduction of the e-learning system Moodle as part of the teaching process is 
related with students’ performance, i.e. the average grade and the average number of admissions to the exams. We also 
examined the same relationship among different groups of students (based on selected individual socio-demographic 
factors). The study included a member of the University of Ljubljana (Faculty of Administration) with public 
administration programmes as our sample in the period from 2008 till 2014. The results of the analysis (using a t-test) 
show a significant improvement in performance (a higher average grade and lower average number of admissions) at the 
different (faculty-, student- and course-) levels after introducing the Moodle e-learning platform. The data show that the 
greatest improvement is seen among students with lower high school grades. The results can serve as important 
guidelines for university management when further investigating how to enhance students’ performance on different 
levels when employing modern ICT solutions in the teaching process. 
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1. INTRODUCTION 

In recent years, e-learning has been becoming increasingly interesting for society and educational institutions 
because it supports the concept of lifelong learning and since knowledge is becoming more and more 
important. This increases the demand for various educational forms and means. Different education 
programmes around the world cater to this increased demand and offer new forms of education that are 
frequently supported by information-communication technology (ICT) (Sulcic and Lesjak, 2009). Moreover, 
technological advances have revolutionized teaching and learning processes (Aristovnik, 2012). Fry (2001), 
for instance, notes that the emergence of new technologies, the rapid expiration of knowledge and training, 
the necessity of just-in-time information delivery, and the need for more cost-effective teaching methods 
have transformed teaching-learning practices. 

In comparison to traditional classroom instruction, the major advantages of e-learning are reducing 
geographical barriers as well as travel and programme overhead costs. The access becomes more flexible, 
from anywhere and usually at any time - essentially, it allows participants to collapse time and space (Cole, 
2000) - however, the learning materials must be designed properly to engage the learner and promote 
learning. On the other hand, there are also many problems that make e-learning ineffective. Many students 
drop out of online courses due to a lack of motivation, instructional design -related factors and learning style 
mismatch, time conflicts with other commitments, organizational support and follow-up on completion 
(Wang et al., 2003). Other problems connected with e-learning are the absence of significant differences in 
acquired knowledge and the unsuccessfulness of e-learning projects (Sulcic and Lesjak, 2009). 
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The effectiveness of e-learning systems has been an important subject of research in the last few decades. 
The opinions on the effectiveness of e-learning are many and very different. There are also many factors that 
influence the effectiveness of e-learning, with some being connected with technology/technics and others 
with people. Upadhyaya and Mallik (2013) claim that each issue of the effectiveness of e-learning cannot be 
treated as a standalone topic - either as a technical matter or as a people issue. E-learning involves interaction 
between people and processes, meaning that has to be treated as a socio-technical system rather than a social 
system considering only the people aspect (e.g. students, teachers and other stakeholders) or a technical 
system only considering the standards and processes aspect (e.g. course content, technology. Learning 
Management System (LMS), content management tools). E-learning is a complex process that depends not 
only on these aspects in isolation, but also the interaction among them. 

The purpose of the paper is to analyse how the introduction of the Moodle e-learning system as part of the 
teaching process is related to students’ performance in the public administration programmes, measured as 
the average grade and the average number of admissions to the exams for each course. We also examined the 
same relationship among different groups of students (based on gender, country region, high school grades) 
and courses (year of study, study programme, chair (faculty’s organizational unit) in which the course is 
conducted). The study included data for the period from 2008 till 2014. The paper explains how introduction 
of the Moodle e-learning system increased the students’ performance on the faculty level and discusses the 
variances among the different subgroups. 

The paper is structured as follows: first, a brief literature review about e-learning and its impact on 
students’ performance is presented. Second, we review the empirical methodology and describe the data. The 
third section outlines the empirical results and the discussion. The final section provides some concluding 
remarks, limitations and directions for future work. 


2. LITERATURE REVIEW 

Since e-learning has been an important and ever more frequently used teaching technique in the past decades, 
there are also many studies on its impact on students’ performance. Delivering instructions that can produce 
equal or even better outcomes than face-to-face learning systems is one of the main goals of introducing ICT 
into the study process (Saba, 2012). But besides the many advantages of this type of study, there are also 
many disadvantages which can reduce the positive impact of modern ICT tools on students’ performance. 
Moreover, in the e-learning process there are also many specific factors involved that are not directly 
connected with ICT but importantly influence the students’ performance, sometimes also in a negative way. 

Sulcic and Lesjak (2009) claim that, although ICT represents a fundamental material condition for 
e-learning, it does not have a statistically significant influence on the effectiveness of e-learning. Similarly, 
Russell (2001) discovered no statistically significant differences between classical and online learning. 
Different authors report varying attrition rates, e.g. from as high as 70%-80% (Dagger and Wade, 2004, 
Flood, 2002) to 20%-50% (Frankola, 2001; Diaz, 2002). These attrition rates can even be higher (between 
10%-20%) than for traditional face-to-face education (Carr, 2000). When measuring students’ performance 
also with other measures, Novo-Corti, Varela-Candamio and Ramil-Diaz (2013) report an increase in the 
performance of students (grades and qualifications) when using the mixed technology of e-learning in 
Moodle and face-to-face lectures. 

According to Ally (2004), the teaching strategies as well as testing and assessment methods are much 
more important than the use of ICT in education. Students’ non-participation in the learning process plays an 
important role in high dropout rates in online learning (Dagger and Wade, 2004). It is therefore very 
important to determine which approaches might increase student engagement (Tyler-Smith, 2006). 

The review of the previous studies raises the dilemma of the validity of such comparisons since the 
statistics on retention and drop outs can be incomparable, unreliable and/or misleading (Hall, 2001; 
Haverilla, 2009; Saba, 2012; Wang, Foucar-Szocki, Griffin, 2003). Researchers acknowledge that the reasons 
for attrition are many and complex and that there are no simple solutions (Berge and Huang, 2004). 

The same is true for students’ performance, measured as average grades and the average number of 
admissions to the exams. There are many factors that influence students’ performance, such as (Chien, 2012; 
Haverila and Barkhi, 2009; Kim and Kim, 2013; Saba, 2012; Upadhyaya and Mallik, 2013; Yukselturk and 
Burut, 2007): learner characteristics (demographic factors, prior e-learning experience, ability and interest. 
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motivation, self-regulation), supporting systems (organizational aspects - e.g. structure and institutional 
arrangements; people aspects - e.g. motivation, training, other stakeholders) and e-learning system (quality 
of contents, tasks (instructor factors), technology - e.g. system reliability, user interface). 

It can be deduced from the reviewed literature that there is a huge research gap concerning the 
contributions of socio-demographic factors of students involved in an e-learning system to their academic 
performance. Nevertheless, a few researchers have focused on selected socio-demographic aspects of 
e-learning, such as Berge and Huang (2004), Yukselturk (2005) and Park and Choi (2009). In general, 
however, individual factors (i.e. socio-demographic, e.g. age, gender, education etc.) seem to have little 
influence on student performance. Indeed, Willging and Johnson (2004) claimed that external factors such as 
family issues, lack of organizational support, and workload are the crucial factors affecting students’ 
performance in an e-learning system (also see Park, 2007). Nevertheless, in our study we tried to fill above- 
mentioned research gap and highlight the importance of a selected socio-demographic background on 
(full-time) students’ performance in the Moodle environment in public administration programmes at the 
University of Ljubljana. 


3. MOODLE: E-LEARNING PLATFORM AT THE FACULTY OF 
ADMINISTRATION (UNIVERSITY OF LJUBLJANA) 

The open-source learning management system Moodle is already widespread in all segments of education, 
from primary schools to universities. It is free, flexible, customizable and basically contains many standard 
features which make it popular. It is available in more than 100 languages. Today, more than 54,000 active 
sites are registered from 225 countries, with millions of users (Moodle, 2015). Kareal and Klema (2006) 
compared particular features of some open-source e-learning systems and found that Moodle is one of the 
most adaptable, which is an essential part of effective education as they pointed out, and mots user-friendly 
learning systems among all the compared ones. Chen et al. (2011) studied the implementation of a Moodle 
course as an extension of a classical learning process at a university and found that for students “the Moodle 
e-learning platform is easy to use and provides a good communication tool, discussion area, group space, 
workspace, and makes learning more interesting”. 

The Faculty of Administration (FA) is part of the University of Ljubljana, Slovenia, that develops 
administrative science through research and education as well as integration into Slovenian and international 
practice. The FA offers two undergraduate study programmes - the University Study Programme in Public 
sector governance (US) and the Higher Education Professional Study Programme in Administration (PS) — 
and one joint undergraduate University Study Programme in Administrative Information Science (UI) 
together with the Faculty of Computer and Information Science. The undergraduate study programme (1 st 
cycle) lasts 3 years (six semesters). Both undergraduate study programmes of the FA, the First Cycle 
Professional Study Programme in Administration and the First Cycle University Study Programme in Public 
Sector Governance, meet the high quality standards defined by the European Association for Public 
Administration Accreditation (EAPAA). The FA also offers the continuation of study in seven postgraduate 
programmes. The study programme is interdisciplinary, combining administrative, legal, economic as well as 
organizational and ICT courses. Consequently, the Faculty is organized in three chairs: the Chair of 
Economics and Public Sector Management, the Chair of the Administrative-Legal Area and the Chair of 
Organization and Informatics. 

The FA began with blended learning in 2005 with eCampus, a payable platform for e-learning. By 
blended learning we mean the current use of the term, namely “combining Internet and digital media with 
established classroom forms that require the physical co-presence of teacher and students” (Friesen. 2012). 
After three years of use, the learning platform was replaced with the open-source Moodle platform, mainly 
due to the user-friendly environment and cost benefits. 

At the beginning, the traditional teaching techniques were combined with an e-classroom on the basis of 
the voluntary decision of lecturers themselves. In the 2010/11 academic year, introduction of the Moodle 
e-learning system as part of the teaching process became mandatory for all courses in the first year of 
undergraduate study, namely 20 to 30 percent of the traditional face-to-face learning process was 
implemented in Moodle. In the next year, blended learning was implemented in the second year of study and 
in the 2012/13 academic year all undergraduate study courses had their own e-classroom in Moodle. 
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At the same time as implementing this combined method of learning, the rales (e-learning policies) on the 
quality of e-classrooms were determined. At the end of each semester, a consultant reviews the e-classrooms 
for e-studies and assesses them regarding their compliance with these rules. The quality assurance of the 
pedagogical work in e-classrooms in accordance with the defined rules is thus regularly monitored and 
necessary improvements are made. 


4. EMPIRICAL RESEARCH 
4.1 Methodology and Data 

In our study we analysed the performance of undergraduate students in two study programmes at the FA- the 
university (US) and the higher education professional study programme (PS), in the period between the 
2008/09 and 2013/14 academic years. We only included the compulsory courses for full-time students in our 
survey since the e-classroom for elective courses was completely mandatory. For each academic year, we 
surveyed all compulsory courses and checked what proportion of their realization (lectures, practical 
exercises) was held in Moodle. Using preliminary analyses, we defined that a course can be understood as 
(also) executed in Moodle if at least 1/5 of its realization is held in an e-classroom (i.e. a minimum 3 weeks 
in a semester of 15 weeks). 

The purpose of the research was to find out if the introduction of the blended learning with support of the 
Moodle platform had increased the students’ performance. We analysed and compared two indicators of 
students’ performance, namely students’ average grades and average required admissions to pass an exam in 
the years ‘with Moodle’ with the (previous) years without it. In the Slovenian higher education system, the 
grading scale ranges from 1 (minimum) to 10 (maximum) with 6 as a minimum passing grade. For our 
analysis, we used Student’s t-test for two independent samples. Table 1 shows the number of students 
enrolled in both programmes in each academic year between 2008/2009 and 2013/2014, where “Moodle - 
NO” means years with only face-to-face classrooms and “Moodle - YES” years with blended learning. 
Notice that in the last academic year all the courses are treated as courses with Moodle since it was 
mandatory for lecturers to have an e-classroom in Moodle. On the contrary, in the first two years (2008/09, 
2009/10) there are no courses treated as courses with Moodle. In the interim period, we have both types of 
courses: face-to-face and blended classroom. It is therefore possible that the same students were selected for 
both the “Moodle - NO” and “Moodle - YES” samples since they attended both types of courses. Notice that 
we only analysed obligatory courses so the students could not choose to be assigned to the “Moodle - NO” 
and “Moodle - YES” samples since they had to follow the lecturers’ decision. 

Table 1. Students enrolled in each academic year - in study years of both programmes (PS - Professional Study 
Programme and US - University Study Programme) 


Academic year 

1 st 

Moodle - NO 

2nd -^rd 

1 st 

Moodle -YES 

2nd ^rd 

2008/09 

PS 

213 

0 

0 

0 

0 

0 


US 

278 

0 

0 

0 

0 

0 

2009/10 

PS 

114 

0 

0 

0 

0 

0 


US 

208 

210 

0 

0 

0 

0 

2010/11 

PS 

157 

0 

96 

133 

0 

0 


US 

163 

191 

138 

148 

175 

0 

2011/12 

PS 

119 

0 

0 

156 

0 

89 


US 

108 

140 

147 

109 

0 

0 

2012/13 

PS 

0 

0 

0 

133 

0 

65 


US 

92 

103 

131 

0 

0 

0 

2013/14 

PS 

0 

0 

0 

110 

0 

67 


US 

0 

0 

0 

93 

80 

82 


Source: Survey, 2014 
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As explained in the above literature review section, students’ performance is influenced by many factors. 
We decided to eliminate the impact of (1) the fluctuation of lecturers and (2) different generations of 
students. Therefore, we only analysed courses where the same lecturer conducted a course without Moodle 
and later with it, both for at least two consequent years. This means we excluded all courses where the 
lecturer was changed. We also excluded courses where only one generation used Moodle or only one 
generation did not use it. Our original data table was therefore reduced to only 14 compulsory courses. 


4.2 Empirical Results 

On the faculty level, the introduction of Moodle is related to a significantly increased student performance. 
The average grade rose from 6.98 to 7.11 (Table 2), making the difference of 0.13 points highly significant 
(p-value: 2.71E-7). The decrease in the average number of admissions needed for the exams is even stronger 
(Table 3) - it went down from 1.73 admissions (Moodle - NO) to 1.30 (Moodle - YES). The difference of - 
0.43 is even more significant (p-value: 7.26E-86). 

Although the introduction of the Moodle platform at the faculty level showed a significant improvement, 
we assumed there might still be subgroups of students or courses where the improvement is not significant. 
For a more detailed analysis, we collected additional data on courses and students (factors). We added data 
about the study programme (US/PS), the year of study (1 st , 2 nd , 3 rd ) and the chair (organizational unit at the 
Faculty) to which a lecturer belongs. From the students’ enrolment form we extracted data on the students’ 
gender and the region of Slovenia where they were born. We only considered two ‘regions’ - “Central 
Slovenia” where the faculty is located - and all other regions in Slovenia as one region (“Other Slovenian 
regions”). In addition, we took the students’ high school final grade into account (four categories). 

Table 2. Average grades for years with Moodle (Moodle - YES) and years without it (Moodle - NO) among different 
factors (study programme, year of study, chair of lecturer; gender, region and high school final grade) with differences 

and corresponding p-values (Sig.) 



Moodle 
NO YES 

Difference 

Sig. 


Programme 

University Study Programme 

6.99 

7.12 

0.13 

3.18E-06 

*** 

Professional Study Programme 

6.93 

7.08 

0.15 

2.09E-03 


Year of study 

1 st 

7.09 

7.20 

0.11 

6.80E-04 

* 

2 nd 

6.82 

6.99 

0.17 

4.06E-05 

*** 

3 rd 

6.89 

6.94 

0.05 

2.40E-01 


Chair 

Economics and Public Sector Management 

6.79 

7.06 

0.28 

4.88E-15 

*** 

Administrative-Legal Area 

7.24 

7.37 

0.13 

1.10E-02 


Organization and Informatics 

6.91 

6.96 

0.05 

1.69E-01 


Gender 

Male 

6.86 

7.00 

0.14 

1.79E-03 


Female 

7.01 

7.15 

0.13 

6.21E-06 

*** 

Region 

Central Slovenia 

6.92 

7.10 

0.17 

8.35E-07 

*** 

Other Slovenian regions 

7.03 

7.13 

0.10 

3.17E-03 


High school final grade 

sufficient (2) 

6.89 

7.01 

0.12 

1.46E-05 

*** 

good (3) 

7.11 

7.17 

0.06 

1.61E-01 


very good (4) 

7.41 

7.45 

0.04 

3.43E-01 


excellent (5) 

7.61 

7.61 

0.00 

5.01E-01 


Together (at the faculty level) 

6.98 

7.11 

0.13 

2.71E-07 



Difference is significant at the following levels: 0.1 (*), 0.05 (**), 0.01 (***) 
Source: Survey, 2014 
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For each factor we calculated the average grade and the average number of admissions to the exams for 
all of their levels separately for the years with Moodle and the years without it. In Table 2 and Table 3 we 
report the averages and differences with corresponding p-values calculated with an independent samples 
t-test. We corrected the p-values with a Bonferroni correction and marked the significant ones with stars. The 
data in Table 2 indicate a highly significant increase in the average grade among the majority of categories. 

The biggest increase in the average grade belongs to the courses in the Chair of Economics and Public 
Sector Management. The average grade rose from 6.79 to 7.06, with the difference of 0.28 points being 
highly significant (p-value: 4.88E-15). We also discovered highly significant increases for students from 
Central Slovenia (increase from 6.92 to 7.10, p-value: 8.35E-7), students from the University programme 
(from 6.99 to 7.12, p-value: 3.18E-60), female students (from 7.01 to 7.15, p-value: 6.21E-6), students with 
high school grades 2 out of 5 (from 6.89 to 7.01, p-value: 1.46E-5) and students in the 2 nd year of study in 
both study programmes (from 6.82 to 6.99, p-value: 4.06E-5). In the other subgroups, the increase in the 
average grade is not significant. On the other hand, it is interesting that the average grade went up in all 
subgroups with the only exception of students with an excellent high school background (grade 5 out of 5). 
We found at least one subgroup with a highly significant increase in the average grade for each factor 
(programme, year, chair, gender, region, high school grades) which helps us identity where the 
implementation of Moodle seems to play an important role in achieving high students’ average grades. 

Table 3 shows a highly significant decrease in the average number of admissions to the exams among all 
categories. A slight exception is the factor ‘high school final grade’. 

Table 3. Average number of admission for years with Moodle (Moodle - YES) and years without it (Moodle - NO) 
among different factors (study programme, year of study, chair of lecturer; gender, region and high school final grade) 

with differences and corresponding p-values (Sig.) 



Moodle 





NO 

YES 

Difference 

Sig. 


Programme 

University Study Programme 

1.76 

1.32 

-0.44 

3.40E-61 

*** 

Professional Study Programme 

1.58 

1.23 

-0.35 

9.16E-21 

*** 

Year of study 

1 st 

1.72 

1.20 

-0.52 

1.05E-75 

*** 

2 nd 

1.76 

1.43 

-0.33 

2.63E-15 

*** 

3 rd 

1.66 

1.44 

-0.22 

2.37E-04 

*** 

Chair 

Economics and Public Sector Management 

1.76 

1.27 

-0.49 

8.16E-58 

*** 

Administrative-Legal Area 

1.43 

1.25 

-0.17 

7.17E-07 

*** 

Organization and Informatics 

2.02 

1.42 

-0.60 

4.44E-28 

*** 

Gender 

Male 

1.78 

1.32 

-0.45 

6.13E-23 

*** 

Female 

1.71 

1.29 

-0.43 

1.90E-65 

*** 

Region 

Central Slovenia 

1.75 

1.32 

-0.44 

2.81E-43 

*** 

Other Slovenian regions 

1.70 

1.27 

-0.43 

3.21E-41 

*** 

High school final grade 

sufficient (2) 

1.80 

1.32 

-0.47 

1.17E-63 

*** 

good (3) 

1.60 

1.28 

-0.32 

4.20E-13 

*** 

very good (4) 

1.40 

1.21 

-0.19 

5.89E-04 

* 

excellent (5) 

1.43 

1.13 

-0.30 

2.32E-03 


Together (at the faculty level) 

1.73 

1.30 

-0.43 

7.62E-86 



Difference is significant at the following levels: 0.1 (*), 0.05 (**), 0.01 (***) 
Source: Survey, 2014 
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Students with an excellent (grade 5 out of 5) and very good (grade 4 out of 5) high school background 
saw a decreased average number of admissions. The drops from 1.43 to 1.13 (excellent students) and from 
1.40 to 1.21 (very good students) are, however, too small to be significant. We can see that those students 
with a better high school background (grades) need fewer admissions than the students with lower high 
school grades. It is, however, encouraging that the Moodle environment helped students with a lower high 
school background reduce the average number of their admissions. Students with sufficiently high school 
grades (grade 2 out of 5) decreased the average number of admissions from 1.80 to 1.32. Although the 
decrease is significant for all factors, the data in Table 3 show a very significant decrease in the average 
number of admissions for the 1 st year of study (from 1.72 to 1.20 with p-value: 1.05E-75) and female 
students (from 1.71 to 1.29 with p-value: 1.90E-75). We can also find highly significant differences in the 
University Programme and courses from the Chair of Economics and Public Sector Management. 


5. CONCLUSION 

The results of our study indicate that the implementation of an e-learning system (Moodle) at the Faculty of 
Administration, University of Ljubljana is related to a statistically significant increase in students’ 
performance, measured as the average grade and the average number of admissions to the exams. We 
demonstrated that the courses from the Chair of Economics and Public Sector Management, female students 
and students with a lower high school background benefitted more from the work in the Moodle environment 
than the other groups of studied entities. Based on our empirical results, we can also conclude that almost all 
subgroups of courses and students reduced the average number of admissions to the exams and increased the 
average grades after the Moodle platform was introduced. Interestingly, the only subgroup of students which 
did not benefit much is the group of students with the highest grades from high school. 

The main limitation of the research is the limited data set (i.e. the reduction of the number of courses 
analysed due to the lecturers’ fluctuation and different generations of students). Besides that, we did not take 
into account other individual factors (e.g. motivation) and external factors (e.g. contents quality, previous 
trainings in Moodle, instructor factors, technological characteristics of the Moodle platform). These factors 
should be the subject of our further research in the near future. Moreover, future research could also broaden 
the scope of the current study, which was restricted to the public administration programme. Finally, 
additional studies should seek to analyse the data from different study programmes in order to increase the 
validity of results for the entire University of Ljubljana. To conclude, the study results can still serve as 
important background material when deciding on the future development of e-learning at the Faculty of 
Administration as well as on the introduction of e-learning platforms at other faculties within the University 
of Ljubljana. The empirical results pointed out the main challenge: how to use the Moodle platform to 
increase the grades of students with the best high school backgrounds and therefore to increase the graduates’ 
capabilities to solve challenges in public administration. 
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ABSTRACT 

Student performance in a Connectivist Massive Open Online Course (cMOOC), featuring daily synchronous web 
meetings was analyzed to explore the possibility of increased participant motivation and confidence. Participant survey 
data and course performance data were compiled, analyzed, and interpreted for 1256 individuals who completed a 
pre-course survey and 54 selected individuals who completed the post-course survey and met the sample criteria. This 
qualitative study adds to the overall understanding of social learning in a blended cMOOC environment. Results suggest 
that Daily Online Task-based Social (DOTS) Learning in large-scale web meetings may increase an individual's language 
learning confidence and personal motivation. 
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1. INTRODUCTION 

Massively Open Online Courses are criticized due to the “Retention Problem,” (Roller, Ng, Do, Chen, 2013) 
demonstrating high numbers of participants in an online MOOC may not promote successful learning 
outcomes. Efforts to provide a more accurate picture of student success have included categorizing 
participants as “Passive participants, Active participants or Community participants,” (Milligan, Allison, 
Margaryan, 2013). In addition, participants in MOOCs often report feeling “isolated and alone” which may 
reduce motivation and confidence, (Bischoff, 2000). In an attempt to increase the number of students 
completing online programs, different designs have been suggested, including social learning opportunities, 
blended asynchronous and synchronous interactions in a connectivist environment or cMOOC, (Siemens, 
2005). 

To help explore these issues further, a Spanish Language cMOOC called “Cada Dia Spanish” was offered 
through the Canvas Network, for eight weeks in the fourth quarter of 2014. The online course was a 
demonstration of how daily synchronous meetings could be applied to a cMOOC, for language learning. 
Over 1700 participants signed up for the course. Study participants were selected based on criteria, described 
below, and resulting in 121 participants self-identified as active participants and 54 individuals selected as 
study participants. Our intention was to explore ways to increase personal motivation and confidence, with 
the assumption that language skill may improve as a result. This study explores two specific questions: 

• Does participant motivation improve through daily, online, task-based, social learning? 

• Will participants demonstrate increased language learning confidence in this cMOOC? 

The qualitative study seeks to determine factors contributing to language learning success (confidence 
and motivation), rather than providing a comprehensive language learning program. 
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2. CADA DIA SPANISH - cMOOC DEMONSTRATION PROJECT 

The design team reviewed various cMOOC models (Siemens, 2005; Littlejohn & Margaryan, 2013) prior to 
the development of a design framework for the Cada Dia Spanish cMOOC. A review of literature focused on 
significant indicators for student success and how social presence and Socialcultural Theory (SCT) affects 
student outcomes, (Ellis, 2003; Lantolf & Thorne, 2006). Some of these same benefits of teacher 
confirmation and social presence may transfer to technically sophisticated online social, instructor-led and 
synchronous learning environments. There are three themes we explored in the development of the Cada Dia 
experimental cMOOC: Connectivist MOOCs, Social Learning, and Task-based language learning. 

2.1 Connectivist MOOCs 

In the connectivist MOOCs, or cMOOC model, learners take a greater role in “shaping their learning 
experiences than in traditional online learning, while facilitators focus on fostering a space for learning 
connections to occur,” (Milligan, Littlejohn & Margaryan, 2013). cMOOCs focus on knowledge creation and 
generation rather than "knowledge duplication" (Siemens, 2012). Participant learners are asked to identify 
individualized learning goals in a self-directed environment. Students are provided a framework to respond 
to various activities at their own pace. 

In a connectivist learning environment, the teacher assumes the role of facilitator, and guides students 
through feedback and encouragement. Although the facilitators are not responsible for imparting information, 
their role to direct students to find, analyze and share knowledge is critically important. However, massive 
courses make individualize instruction difficult, and synchronous online meetings have provided a method of 
maximizing facilitator to student interaction, (Henry, 2011). The cMOOC design removes the traditional 
“Instructor or Teacher” role and strives “to integrate instructional strategies that connect learners with each 
other in meaningful and authentic ways,” (Stewart, 2013), which will hopefully reduce feelings of isolation 
and loneliness. 

2.2 Social Learning 

Sociocognitive theorists describe social learning as an interactive group process in which learners actively 
construct knowledge and then build upon that knowledge through the exchange of ideas with others. The 
Social Learning model relies on the interaction of three important techniques: social presence, teaching 
presence, and cognitive presence, (Garrison, Anderson, & Archer, 2000). 

2.2.1 Social Presence 

Social presence, in an online learning environment via computer -mediated communication (CMC) represents 
the degree of awareness of the other person in a communication interaction. If there is no “awareness” of 
another student or facilitator, the individual will feel alone. Several studies have observed that there is 
increased social interaction when the instructor “has not forgotten the importance of the human touch” 
(Pacansky-Brock, 2014). 

2.2.2 Teaching Presence 

Teaching presence is an understanding of the presence of an instructional path, designed by a facilitator, 
teacher or designer. This presence may be observed in videos, activities and communications from the 
facilitators. Teaching presence is strengthened through daily email correspondence, and synchronous web 
meetings. 

2.2.3 Cognitive Presence 

The third element, cognitive presence, is “the extent to which the participants in ... a community of inquiry 
are able to construct meaning through sustained communication” (Garrison et al., p. 89). Online Learning 
environments that are instructor-led, combined with synchronous web conferencing may be able to take full 
advantage of the interaction between social presence, teacher presence and cognitive presence. 
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2.3 Task-based Language Learning 

Task-based language learning (TBLL), also task-based language teaching (TBLT) or task-based instruction 
(TBI) employs the use of common language exchanges, (Ellis, 2003; Hismanoglu & Hismanoglu, 2011.). A 
key component of TBL includes a shift from teacher-produced material to student-produced material. 
Students are asked to share learning experiences, rather than the teacher directing learning experiences. 
Since conjugations, vocabulary and grammatical syntax may be easily found online, facilitators guide 
students through the task framework, and challenge them to improve their responses (Stroud. 2013). The 
language facilitators do not teach, rather they encourage student to student feedback and peer support. 

Participants are encouraged to help each other revise and improve their responses to the task-based 
problems. Some have suggested that the approach is not pedagogy, but “paragogy,” relating to peer 
production environments (Corneli & Danof, 2011) including the co-creation of ad-hoc spaces for dialogue 
and support. Facilitators provide quality criteria for knowledge creation and generation. The notion of 
Learner Generated Content (Perez-Mateo, Maina, Guitert, & Romero, 2011) is associated with learners’ 
productions in Web 2.0 and networked environments. This type of social learning experience facilitates 
informed but personal views on topics and how they contribute to knowledge construction. 

2.4 Cada Dfa cMOOC Course Design 

The Cada Dfa demonstration project cMOOC was designed to address weaknesses identified in other MOOC 
models, such as participant isolation and loss of personal motivation. The experiment was designed to 
improve individual motivation and confidence through Daily, Online, Task -based. Social or “DOTS” (Henry, 
2011) language learning, with daily online synchronous web meetings. The language learning tasks in the 
Cada Dfa Spanish cMOOC were authentic activities that a friend may share with another friend. They were 
simple activities, with opportunities for increasingly sophisticated responses. Participants were invited to 
share their activity in a synchronous web meeting. These activities are included below and in figure 1 : 

• Identify a song in the target language and share the lyrics. 

• Identify a word and share a sentence with the word. 

• Identify a short story and read the story. 

• Identify a verb, conjugation and sentences using the verb. 

• Identify a conversation tip, and share the communication strategy 

• Compose a short description of a trip you have taken or would like to take 

• Compose a short description of your family and friends. 

These activities were posted in the Canvas Learning Management System, for asynchronous review and 
revision. Participants were invited to share their activity in a synchronous web meeting. Synchronous web 
meetings were held at Noon EDT UTC-4 hours. Over the eight week program offering, there were 40 
weekday meetings and two special Saturday meetings. 


107 



ISBN: 978-989-8533-40-1 ©2015 



Figure 1. Cada Dla Spanish cMOOC task-based activities. These activities were posted in the Canvas LMS, then 
participants were invited to share the activities with native language facilitators in a synchronous web meeting 

2.5 Qualitative Study Method 

Qualitative analysis of individual participant motivation and confidence levels was observed using 
participation data in the eight week cMOOC, and through pre-course and post-course feedback comments. 
Motivation was observed through increased activity, as well as self-reporting levels of personal confidence. 
The methods for observing increase (or decrease) in personal motivation included the end of activity personal 
assessment and survey, attendance at the synchronous daily web meetings and a willingness to volunteer to 
present activities in the web meetings. 

2.5.1 Participant Surveys 

• Pre-Course Survey - “Welcome to Canvas Network” - Survey. Administered online to enrolled 
cMOOC participants. 10 quantitative questions and 2 text response questions. (N=298). 

• Post-Course Satisfaction Survey - User Experience Survey. Administered online to enrolled 
cMOOC participants. 10 quantitative questions and 2 text response questions. (N=54). 

• Activity Assessments and Survey - These surveys appeared after the participant completed an 
activity. They included a total of six questions: Four questions, related to motivation and 
confidence and two randomly selected language skill assessment questions. The post-activity 
assessment was administered online in Spanish with English translations. 

2.5.2 Study Sample 

The qualitative study sample was determined by selecting participants who completed all surveys and who 
participated actively in the cMOOC for the entire eight-week period. The course was offered through the 
Canvas Network and advertised through social media, Google Adwords and Facebook. Over 1700 
participants signed up for the online class and expressed an interest in the daily web meetings. From this 
initial pool, 1256 completed the pre-course assessment and survey. Participants were asked to self-identify as 
an “Observer, Active Participant, or Passive Participant.” Only self-identified “Active Participants” were 
included in the qualitative study. Passive Participants were not interviewed or surveyed, since participation 
is critical in the analysis of increased or decreased confidence and motivation. 
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We were fortunate to have a large group of willing participants, however many only submitted one or two 
activities. From the enrolled participants, only a small number participated to the extent we would categorize 
as “Active Participants.” 

Table 1. Cada Dla Spanish cMOOC Participant Activity 


Activity Category Submission 

Total Asynchronous Tasks Posted 919 

Total Task Assessment Responses 684 

Total Synchronous Participants* 321 

Total Synchronous Presenters* 118 


* These represent unique participants, who would have likely participated in more than one activity. 

In addition to selecting participants who completed survey responses, we looked at participant online 
learning activity. We did not include time as a metric of learning activity. It has been suggested in other 
studies that the length of time studying a language would likely improve skill, as well as confidence 
(Vesselinov and Grego, 2012), but in an online environment, it is difficult to measure active time. Study time 
is difficult to document based solely on the time a participant started an activity, until the participant finished 
an activity, since they would likely be reading and studying outside of the application as well. Therefore we 
measured participant’s learning activity based on completed activities, rather than time spent in the online 
learning environment. 

The participants identified for further qualitative analysis were selected based on these criteria: 

• Completed all study surveys 

• Willingness to participate in the qualitative study 

• Identified as an “Active Participant” 

• Completed all seven learning activities 

• Participated in at least ten web meetings 

• Presented an activity in at least one web meeting 

• At least 18 years of age 

• Native English language speaker 

This selection process resulted in a total of 54 participants to be analyzed for patterns of engagement and 
self-reporting. We were interested in how the active participants perceived gained or lost confidence and 
motivation over the eight-week program. 


Table 2. Cada Dla Spanish cMOOC Demonstration Project Participants 


Participants 

Number 

Total cMOOC Participants 

1703 

Total Asynchronous Participants 

647 

Total Synchronous Participants 

321 

Self-Identified “Active Participants” 

118 

Study Participants selected 

54 


3. RESULTS 

3.1 Participant Confidence and Personal Motivation 

Does the cMOOC format, coupled with daily web meetings for participant presentation increase confidence 
and personal motivation? For the purposes of this study, we defined “Participant Motivation” as the change 
in the number of asynchronous actions such as activity submissions, assessment responses, and synchronous 
actions through web meeting attendance and web meeting presentation) s). After participant data was stripped 
of personal identification, each response was weighted based on the level of confidence indicated. For 
example, “I am not confident” was weighted 1 and “I am very confident” was weighted as a 10, using a 
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Likert Scale. We asked participants to self-report “hours studying each week,” although we did not track 
actual online hours of activity. We also looked at actual participant “Activity,” which reflected weekly 
activity submissions and participation in synchronous events over the eight week program offering. A 
specific individual (coded by participant number) was compared for each activity survey response. 
Individual responses were plotted on Figure 2, showing the general trend individual confidence and 
motivation over the entire experience. 


Table 3. Motivation Survey Response Weighting 


Question 

Weighting 

Q7: 0 to 1 hour 

[1] 

Q8: 1 to 3 hours 

[2] 

Q9: 3 to 5 hours 

[5] 

Q10: 5 to 10 hours 

[8] 

Q1 1 : More than 10 hours 

HO] 

Q12: 1 am not confident 

[1] 

Q13: 1 am a little confident 

[2] 

Q14: I am more confident 

[8] 

Q15: 1 am very confident 

[10] 

Q16: 1 want to speak fluent Spanish 

[10] 

Q17: 1 want to speak Spanish well. 

[8] 

Q18: 1 want to speak a little Spanish 

[5] 

Q19: 1 am just observing 

[1] 


Confidence Rating over Eight Weeks 

90% 



10% 

0% 

Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7 Week 8 


= = Activities ““-Hours Objective Confidence 


Figure 2. Confidence Rating Over Eight Week Period 

3.2 Selected Participant Feedback and Comments 

• Participant 166852 Volunteering was helpful for me to get rid of my fear of speaking in Spanish. I 
learned better from the activities I shared. 

• Participant 170799 I feel the only way to learn something like a foreign language is to immerse 
yourself. When you're not exposed to Spanish speakers regularly, this is the next best thing. 

• Participant 201085 I volunteered because it was a great way to practice the language, and interact 
with others as well. I know basic Spanish, I mostly needed help speaking the language. 
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• Participant 172152 The feedback for my mistakes was immediate and really upped my learning 
curve. 

• Participant 168080 I was able to choose the level of my participation, meaning if a question was 
too difficult for me to answer, I could try a more simplistic answer and, thus, still participate. 

• Participant 151464 I had an opportunity to learn from the feedback given to more advanced 
students. 

• Participant 192840 "he expertise of the Facilitators staged an obviously "safe" place to make 
mistakes. Something most of the other students mentioned to me as being one of the things they 
appreciated most. 

• Participant 121560 Humor was included, which created a more enjoyable and memorable 
experience. 

• Participant 184261 I liked the mild, almost passive way I learned new things. It was a refreshing 
departure from the traditional teacher-lectures-students approach. 

• Participant 209679 This course has been both fun and interesting. The Live Meeting and 
conversing with students all over the world is inspiring. Learning another language is helpful in so 
many ways both personally and professionally. It opens new windows to the world and in so offers 
new perspectives to a person's life. 

• Participant 197501 Through this course I am much more motivated to learn than before. The 
activities have helped me a lot improve my level of proficiency. 

• Participant 183102 It has given me more confidence and expanded my ability to understand spoken 
Spanish. I have also learned to write sentences in Spanish extemporaneously. 


4. CONCLUSION 

Our intention was to determine if the cMOOC format, coupled with daily web meetings for participant 
presentation, would lead to an increase in motivation and confidence. The participant commentary indicates 
that the cMOOC format did improve motivation and confidence, as defined in this study. However, the 
responses were largely anecdotal and the data results were restricted to participants with high engagement. 
Further study is needed to demonstrate relationships between social online meetings, increased language 
skill, perceptions of confidence, and persistence of motivation. Research questions related specifically to this 
study could include: 

• Does individual goal identification improve motivation or language learning outcomes? 

• Does participation in live meetings increase participation in other online programs, (such as 
Duolingo, Busuu, WeSpeke, SpanishDict, etc.) 

• Does the existance of synchronous opportunities in a cMOOC increase retention? 

A recommendation resulting from this study is to encourage daily, social interactions through language 
learning tasks shared with facilitators and participants in web meetings. Other studies have found that 
students who lack motivation or confidence in their study of Spanish, often "studied very irregularly, e.g. a 
few hours in the beginning of the study followed by long periods of inactivity," (Vesselinov and Grego, 
2012). If motivation and confidence can be increased through daily social task-based learning, it would 
follow that language comprehension may increase as well. The approach used here, daily large-scale web 
meetings, using structured task-based activities, may increase social engagement and motivation to reach 
individual language learning goals. 
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ABSTRACT 

Teaching students to create computer games has become a common practice in both K-12 and tertiary education to 
introducing programming concepts, increasing student engagement, and recruiting majors and minors in technology 
fields. This study describes a project where first-year college students in an introductory technology concepts course use 
a visual game creation tool to develop original games to play on their computers and mobile devices. The paper argues 
that the process of making an original computer game develops digital literacy skills and provides an authentic learning 
experience as students collaborate to create, publish, and deploy interactive games that they can play on their computers 
and mobile devices. 


KEYWORDS 

Game Development, Digital Literacy, Authentic Learning. 


1. INTRODUCTION 

Many educators have turned to game development as a way to engage students with programming and 
technology concepts. Creating games to play on computers and mobile devices requires specifying rules of 
play, creating multimedia, posting the files for distribution on multiple channels, and navigating operating 
system features. This paper contends that completing these tasks allows students to demonstrate many digital 
literacy proficiencies and provides an authentic learning experience. 

1.1 Digital Literacy 

The need for today’s students to be digitally literate is growing, and educational organizations worldwide 
have responded with new curricula to teach digital literacy skills. In the United Kingdom, the Department of 
Education has teamed up with Microsoft to produce a National Curriculum Guide for K-12 students outlining 
learning goals. Throughout their education, this curriculum provides opportunities for students to apply 
principles and concepts of computer science, including abstraction, logic, algorithms, and data representation; 
to analyze problems computationally; to evaluate and apply information technology, and become responsible, 
confident, and creators of information and communication technology. (Department of Education, 2014) In 
the United States, a National Science Foundation Grant has sponsored the development of new curricula for a 
higher education advanced placement computer science principles course to encourage computational 
thinking and problem solving skills. (College Board, 2015) The course, to launch in, 2016 will “introduce 
students to creative aspects of programming, using abstractions and algorithms, working with large data sets, 
understandings of the Internet and issues of cybersecurity, and impacts of computing that affect different 
populations.” 

The National Computing Curriculum identifies pathways to digital literacy through a student’s 
understanding of algorithms, programming and development, data and representation, hardware and 
processing, communication and networks, and information technology. (Department of Education, UK, 2014) 
Teaching students to create their own computer games has become a popular approach to introducing 
programming concepts to introductory computer science and technology students. (Mohammed & Mohan, 
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2010), (Guzdial, 2015), (Bayliss, 2009). (Elliott, 2007). From an instructional perspective, creating games 
offers opportunities for teaching programming and software development skills on many levels. Games are 
engaging, allow students to learn by playing, and enable students to turn their ideas into real-world 
applications. In addition to promoting algorithmic thinking and problem solving, creating their own games 
presents students with opportunities to be creative, and demonstrate their proficiency as able participants in a 
world based on digital literacy skills. 

1.2 Authentic Learning 

Authentic learning pedagogy enables students to engage in realistic tasks using real-life resources and tools, 
and offers opportunities for students to learn while taking on the roles of professionals dealing with actual 
problems that arise. (Herrington, 2003). (Herrington, Parker, & Boase-Jelinek, 2014) Authentic learning 
environments have real-world relevance, are not well-defined, and require investigation of a task over a 
period of time, and from different perspectives. They encourage collaboration and reflection, and span 
several subject areas. Authentic activities integrate with each other such that they reflect real world 
assessment, and allow competing solutions with a diversity of outcomes. They “engage learners in the work 
of professionals” (Elliott, 2007, p. 34). The instructor fashioned a game creation exercise and contest for 
students to learn about software development, and required students to publish their games on two different 
distribution channels, each with its own set of rules and standards. The adaptation of industry-standard 
procedures as a classroom exercise provides a new way to engage students in authentic learning as they build 
digital literacy skills and complete the tasks necessary to create and share their games with the world. 

1.3 Research Questions 

This study shares preliminary results after offering a game development experience to beginning information 
technology students. Academic goals of the exercise were to provide an authentic learning experience that 
enabled students to apply digital literacy concepts and skills, and empower t to create computer games that 
they can play on their own devices. 

The study addresses the following research questions: 

• Will students with little or no previous experience in developing software succeed in creating 
games? 

• How does creating and deploying games provide for authentic learning as students develop and 
apply digital literacy skills? 


2. CREATING GAMES TO DEVELOP DIGITAL LITERACY SKILLS 

This study introduced students to the use of Construct 2 (Scirra.com, 2014), a gaming development tool that 
enables users to create advanced 2-D games without the use of any programming language. Construct 2's 
game creator allows users to export their games as web apps by generating HTML5, CSS3, and JavaScript 
files, and also to generate native app packages for Windows Phone, iOS, and Android, that can be published 
on the Windows, Apple, and Google Play app stores and marketplaces. 

Suitable for beginners who have little or no experience developing software applications. Construct 2 
enables students immediately to take on the role of game developers. They design backgrounds, layouts, and 
objects (characters, platforms, bullets, sounds, and so on) to be used in the game, and specify their properties 
and interactions during game play. Figure 1 shows the properties of a player object in a 2-D platform game. 

Construct 2 uses an event-driven syntax to describe the interactions between these objects. As shown in 
Figure 2, a rule describes an end-of-game scenario: “When the player collides with the red dot object (Sprite 
9), destroy the player and set Fives back to 4 (because the game is over).” The developer would specify 
similar rules for game play based on input from arrow keys, touch, or tapping buttons on the screen. 

A game must pass several rigorous tests before an app store will accept it for publication. The developer 
must specify the game’s orientation (portrait or landscape), test the game in an emulator to ensure it functions 
correctly on a variety of devices and screen sizes, register the game with the app store by finding an available 
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name, provide a series of screenshots and icons at different sizes, package all of the necessary files app, and 
upload them to the store. Students found this process to be quite complicated, though none of the individual 
steps involved were complicated on their own. 


Layout 1 xj Layout 2 Layout 3 Event sheet 1 


Properties 9 

'5 *1 
’= z* 

ffi Position 99, 282 

0 Size 46, 43 

G Instance variables 

Add / edit Instance variables 

B Behaviors 
0 Platform 

Max speed 330 
Acceleration 1500 
Deceleration 1500 
Jump strength 650 
Gravity 1500 

Max fall speed 1000 
Default controls Yes 
Initial state Enabled 
0 Wrap 

Wran to I avnut 

Figure 1 . Specifying properties and behaviours of the Player object, 
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Figure 2. Describing a rule in Construct 2. 


Figure 3 shows a student's game published to the Windows Store. 



Meeblo2 



Free ★★★★★ i 

Published by CIS Sandbox 
Copyright 2014, Wyatt 

Category Games / Adventure 
Approximate size 5.7 MB 
Age rating 12 + 


i Windows Store 


More apps by CIS Sandbox > 

B Dodge Dat Raind... 
Free 

★★★★* No rating 

□ Crazy Miley Game 
Free 

★ No rating 


Score: 11 



Description 

Join Meeblo on his adventure to reclaim his stolen orbs! Destroy enemies, navigate complex levels and unlock new levels and 
abilities in this platform-style adventure. Use Meeblo's shift ability to circumvent obstacles and attack enemies. 


Figure 3. A student's game published to the Windows Store. 

When preparing an app for the store, the developer may specify pricing information if the app were to be 
offered for sale (for the sake of this assignment, students could not sell their apps, as doing so would require 
additional complexities related to setting up merchant accounts); identify an age-appropriate rating, and 
supply a complete description and game play notes for game testers. 
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These real-world tasks required to submit a game for publication demonstrate a student's basic digital 
literacy skills. Students must be adept at interacting with files, creating graphics, converting files from one 
type to another, and using web-based tools to accomplish a sophisticated task. Completing these steps also 
gives students additional appreciation of how apps, such as the ones they download routinely to their devices, 
are published and deployed via an app store. 


3. METHODS 

This section presents the research context, data, analysis, and discussion. 

3.1 Context 

The research context was a required course for first year introductory technology students. The study took 
place in three sections of IT 101 (Introduction to Information Technology and Computing Concepts), an 
introductory IT course required of all first year students at Bentley University, a business university in the 
United States. IT 101 teaches digital literacy skills and covers information technology topics, including 
making use of laptops, productivity and application software, the World Wide Web, computer components 
and mobile devices, developing web pages with HTML, operating systems, the Internet, image and video 
formats, and wireless networking. Three academically different sections of the course taught by two 
different instructors participated in the project. 

• An Honors section, for students enrolled in the honors program, chosen because of their high 
scholastic abilities, 

• An Accelerated section, offered to students who self-selected to be this section because of their 
interest in technology, and 

• A standard section open to all students. 

Each section met twice a week for 75 minutes each session. Microsoft sponsored the gaming project and 
provided an academic technology evangelist to lead the training sessions in class. The same Microsoft 
academic evangelist facilitated the Construct 2 trainings to students in all three sections, and the instructors 
assisted students during the demonstrations and exercises in their classrooms. 

Students installed Construct 2 on their laptops prior to the training. Teaching students to use the software 
in the classroom required significant support from the Microsoft evangelists who created classroom resources 
as well as trained tutors to assist students in developing and deploying their games outside of class. 

At the first training, students followed a printed tutorial as the presenter demonstrated each step to create 
a simple 2-D shooter or platform game. Game elements included identifying a player, shooter, layouts, user 
interface elements for keyboard and touch input, detecting collisions of objects, and creating and destroying 
objects. These elements are common to many 2-D games. 

The second session showed how to include advanced features, such as adding music, keeping score, and 
transitioning between different layouts for a start screen and a game over screen. Students also learned how 
to package their games for deployment to the web and to the Windows Store. 

For their assignments, students could create their own original game, or adapt the game they worked on in 
class and add new features such as sound, score keeping, and end-of-game logic. They could work 
individually or collaborate with a partner in designing and developing their games, but students had to create 
their own individual games (which could be similar) for this assignment. 

Because Microsoft sponsored this gaming training and contest, and because an academic license through 
Microsoft DreamSpark allows students to publish apps to the Microsoft Windows store without the need to 
purchase a developers license, this project instructed students to publish their games to the Windows store. 
(Apple and Android developers must pay an annual license fee to publish apps in those app stores.) 

A Microsoft evangelist, tutors in a university computer lab, and students who had taken the course during 
a previous semester were available to help the current cohort of students with their games during additional 
"studio time" sessions outside of class. As an additional incentive, Microsoft donated prizes to be awarded to 
the winners at a games party at the end of the project. 

Upon the due date, tutors and former students played and reviewed all of the games from all of the 
students in the current semester, evaluating them on playability and originality, and whether a game met a 
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checklist of required features. From these reviews, the top 7 games were announced. All students then were 
invited to a games party where they could play all of the games and vote for their favourite ones. The winners 
were announced, and prizes awarded. 

3.2 Data and Analysis 

57 of 85 students (37 male, 15 female, 5 prefer not to say or unanswered) from three sections of IT 101 
voluntarily participated in an online survey within two weeks after the gaming assignment ended, to share 
their feedback about the experience. 

Of the 57 responses, 44 were complete and used for this analysis. The remaining 13 students did not 
answer all of the survey questions used in this study. 

Results show that these students use their computers and devices to accomplish daily tasks: 34% of their 
time online is spent for homework, learning or professional reasons, 14% keeping up with news, 20% of their 
time is spent communicating or on social networks, 22% of their time they use the web for entertainment 
including music, videos, and games, and 9% of their time is spent online shopping, banking, or other business 
tasks. 

Despite their being tech savvy, 83% responded that they had not created software applications of any kind 
prior to the game assignment. 17% of respondents had created software applications in high school, including 
simple Java, Visual Basic, or C# programs, web pages developed in HTML, CSS, and JavaScript, and games 
using Flash or other game creation tools. None of the students had used Construct 2 previously. These results 
suggest that creating games to play on their computers and mobile devices was a new experience for most of 
the students, and certainly, deploying them to two platforms was new for all of the students. 

3.3 Results and Discussion 

As shown in Figure 4, survey results suggest that the gaming assignment instilled confidence, captured 
interest, provided authentic skills in writing programs and apps, and allowed students to reflect on their 
successful results. 

Attitudes toward Programming 

25 



I would like to learn I'm sure I could learn I wish I could develop It would be helpful to 
to write computer to write programs my own software know how to write 
programs. applications software 


■ Strongly Disagree ■ Disagree ■ Neutral ■ Agree Strongly Agree 
Figure 4. Student attitudes toward programming and creating software. 

Said one student, "Games can be easier to make then one thinks." Several student comments reflected the 
sentiments that "game development is a lot of hard work but it is very rewarding to see a finished project 
after putting many hours of development into it. 1 learned to have a final vision but also think step-by-step to 
make that vision come to reality." Other students got lucky in the process of creating their games: "if you 
miswrite part of a code, a function you did not expect occurs... You gotta go with the flow if something 
unexpected works." 
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3.3.1 Game Development as Authentic Learning 

"An authentic, challenging task is the starting point. Authentic tasks are completed for reasons beyond 
earning a grade. Students also see the activity as worthwhile in its own right." (Means & Olson, 1994, p. 15) 
The survey asked students their opinions of the gaming assignment as an authentic learning experience. 

"Authentic learning requires students to reflect on their experiences and draw conclusions from their 
findings." (Herrington, 2003). In their reflections, many students remarked that the assignment was "real", 
gave them a taste of building apps, developed and applied their IT 101 skills, and most were pleased with the 
games the created, as results suggest in Figure 5. 

■ 

Attitudes toward Game Development Assignment 

30 



Creating a game I am pleased with let me use many developed my skills was real gave me a taste of 
was an interesting the quality of the skills I learned in IT in thinking logically building apps 

way to apply IT game that I 101 

concepts created 

■ Strongly Disagree ■ Disagree ■ Neutral ■ Agree Strongly Agree 

Figure 5. Attitudes toward game development assignment as authentic learning 

Results were mixed as to whether students found the experience authentic based on the survey aspects 
above. Of those who were not neutral in their responses, most agreed or strongly agreed that it gave them a 
taste of building apps and made them want to learn more about programming. In an open-ended response, 
students commented that they learned to use Construct 2 and Brackets (an HTML editor), and how to upload 
their games to the Microsoft store. They learned "to do in depth research on a topic and how to make my own 
game," that solving problems by walking through them are the best methods to solve computer issues," and 
one claimed that the assignment allowed the student to "implement higher thinking and creativity into an IT 


aspect." Another student remarked, "The IT field is a lot harder than it look to be from the outside - but in the 
end you've always got an opportunity to create something beautiful." For many students, creating a game was 
the most technically complex task they had ever completed. 

Table 1 summarizes ten characteristics of authentic learning environments (Herrington, 2003), and 
describes how each is manifest in the Gaming Assignment. 

Table 1. 

Authentic Learning Characteristics as Evident in the Gaming Assignment 

Authentic 

Learning 

Characteristic 

As Evident in the Gaming Assignment 

Problem has real- 
world relevance 

Students create a game that runs as a web app or a Windows app. Students 
see the relevance of creating software that runs on their computer or mobile 
device. The exercise mirrors the requirements of a real-world activity. 

Problem is open- 
ended 

Students have the flexibility to use the tools they know in order to design 
their games, and must apply the concepts learned during the classroom 
trainings or perform research to find additional tutorials or examples on 
which to base their own games. Students must ascertain the effectiveness of 
their choices as they try to implement their own ideas to modify existing 
games or create new ones. 
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Task to be 
investigated over a 
sustained period of 
time 

The gaming assignment runs over a period of four weeks, so students have 
time to learn basics, explore alternative solutions, get support from tutors and 
peers, and realize the trajectory of their progress. The project is divided into 
smaller, manageable steps that allow students to reach these milestones. 

Allow for exploring 
a task from 
different 
perspectives 

Students involved in the project are in their first year of college, many have 
not done any programming or development before. Students with a more 
technical acumen were able to add complexities; others looked at specific 
tasks or features such as adding score keeping capabilities, and were able to 
apply the same steps they did in class example to their own games. 

Provide for 
Collaboration 

While students were required to create their own games, they were 
encouraged to work in pairs to share ideas, and review each other's work. This 
proved to be effective as students were able to support each other in their 
learning, show each other how to accomplish various tasks that they 
remembered that their partners did not. Students used appropriate online 
collaboration tools to share screens, graphics, or other files with each other. 

Provide an 
opportunity to 
reflect on the 
experience 

The goal of the exercise was to learn about creating software by developing 
an actual game, and then deploy it to two different distribution channels. By 
developing, debugging, and playing their games they reflect on the logic and 
scenarios they implemented, and how the steps they specified produce the 
desired game play. 


By playing their games and showing them to others, students had an 
opportunity to talk about their work, and reflect on ways that they might want 
to improve it in future revisions. 

Seamlessly 
integrate with 
assessment 

Not only the instructor determines the quality of a student’s game; an 
independent evaluation takes place through the process of submitting the 
game to the Windows store. Each step in the submission process requires that 
the correct files are in place, and descriptions and documentation follow 
specifications. Microsoft testers play each game prior to accepting it on the 
store, testing for touch input and other features. 


Games accepted to the store received an automatic 'A' because they passed 
these stringent requirements. Grades for this assignment considered the 
playability, difficulty, and originality of their games, along with the status of 
their game's acceptance on the Microsoft store. 

Create polished 
products in their 
own right 

The games that students created were completely functioning software apps 
that they could run on their computers and mobile devices. Many featured 
touch input, music, score keeping, animations, and other characteristics found 
in real games. Not only did students create games, they had to deploy their 
games correctly so others could access them on their devices. 

Allow competing 
solutions and a 
diversity of 
outcomes 

While all students completed the same in-class training, the games they 
created were all different. Even variations on the same in-class example 
generated a variety of results and outcomes. A games contest gave added 
incentive to produce a high quality final product. 


4. CONCLUSION 

This study presents preliminary results based on one semester's implementation of a gaming project. 

Future research will aggregate survey results over a period of several semesters to further study how the 
process of creating and deploying games for play on computers and mobile devices encourages digital 
literacy skills and impacts student learning. 
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Initial findings suggest that students found the gaming assignment offers an opportunity for students with 
no prior programming skills to create software within a controlled and supportive environment. It allows 
them to demonstrate their understanding of coding principles, including identifying objects and interactions, 
and that creating software requires a developer to specify exact instructions for the computer to follow. 

Students learned to represent different types of data and information (scores, sounds, characters in their 
games) in a digital context; and to design games for multiple devices and input sources, from touch to 
keyboard. The assignment provides students an opportunity to reflect upon their games and consider 
enhancements for future revisions. They also must consider ethical implications (suitability of content, 
amount of violence in games, and use of open-source sounds and graphics) when building their games. The 
exercise allowed students to experience the role of a game developer and at the same time, create apps for 
their computers and mobile devices. 
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ABSTRACT 

Over the last decade, web has changed the way that educational procedures are delivered to students and has brought 
innovative learning technologies and possibilities that were not available before. The Web has evolved to a worldwide 
platform for collaboration, sharing and innovation, constituting what is called Web 2.0. Social media are an emerging 
part of Web 2.0 and have great potential to be used in education. The efficient integration of social media and Web 2.0 
tools in educational systems and courses’ curriculum mainly depends on tutors' abilities and experiences. However, in 
many cases, tutors are not familiar with social media and with many useful Web 2.0 tools and so fail in utilizing them 
into their courses. In this paper, we present an educational system that aims to assist teachers to develop e-Learning 2.0 
knowledge and skills and thus turn them from just in-class trainers to skilled e-tutors. We present the educational courses 
and the learning approaches followed to assist tutors in applying web 2.0 tools and the social media in their courses for 
the best benefit of the students' learning. The educational system provides courses and learning material, and assists the 
tutor to learn how to utilize social media and web 2.0 tools in classroom or in distance learning with the aim to organize, 
monitor and share learning content, attract students’ attention with entertaining activities and motivate them to become 
more active and collaborate in educational activities. The evaluation study conducted reports very promising results. 


KEYWORDS 

Continuing professional development, training tutors, social media, Web 2.0 


1. INTRODUCTION 

The advert of the web has changed completely the way that eLearning and educational procedures are 
delivered to students and have brought new innovative learning technologies and possibilities. Web 2.0 
technologies are becoming very popular in the everyday lives of students and as a result, teachers and 
education designers have begun to explore their use in formal education. The growth of Web 2.0 and the 
rising of social media have transformed the way that people communicate and collaborate and have the 
potential to enhance the learning efficiency of the educational systems and eLearning procedures (Bennett et 
al 2012). Indeed, social media platforms can become important disruptive technologies for building cutting 
edge models of management education (Thomas et al 2012). In this line, the emergence of web 2.0 has led to 
a new concept known as eLearning 2.0 referring to the use of the social media and the technologies of Web 
2.0 in the relatively unchanged institutional framework that characterizes education (Keats et al. 2007) 
(Lwoga 2012). When applied in education area, Web 2.0 technologies emphasize on students' active 
participation, the student-centered generation of content and collaboration and seem to fit very well with the 
kinds of creative and critical activities associated with all levels of education, with the ways that we know 
students leant through exposure to multiple perspectives, and with the communication and teamwork skills 
wanted for students to develop (Bennett et al 2012). So, the incorporation and utilization of web 2.0 
technologies in education is vital. 

When incorporating any kind of media into courses’ curriculum, educators should consider the construct 
of the course, the technologies used and the pedagogical approaches followed to design and deliver the 
desired learning activities (Blaschke 2014). While eLearning constitutes a commonplace form of education, 
the web tools and applications that are used are mainly first generation web tools rather that second 
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generation, WEB 2.0 tools (Connolly et. al 2011). A reason for this is the tutors' and students' perceived 
complexity of the tools and the lack of training for using them properly. The tutors in many cases are not 
familiar with the trending social media services and the available web 2.0 tools and fail in utilizing them into 
their courses. It is well pointed that tutors need to constantly update their knowledge and capabilities 
regarding the utilization of new technologies in education. However, in many cases, the lack of tutors 
training and familiarity with the new web technologies prevent their educational utilization and development. 
Indeed, in many cases tutors are not familiar with the new web 2.0 technologies and are not aware of what 
they offer and how they can be utilized in education for the best benefit of their students’ learning. 

In order to assist tutors gain skills on the new Web 2.0 technologies we have designed and developed an 
educational system and created various courses and training scenarios. The system aims to broaden the 
e-skills and competencies of teachers, trainers and tutors and help them to develop adequate online training 
practices for effective distant tutoring activities at the workplace and on the go without time and distance 
barriers. In this paper, we initially present the educational system developed, the courses designed and the 
educational scenarios formulated that aim to help the tutors get familiar with various social media and web 
2.0 technologies and platforms and utilize them in the courses they teach. The online courses are designed to 
assist tutors in applying social media tools and techniques in education and also learn how they can integrate 
the social media in their courses’ curriculum. The utilization of social media could enhance the interaction 
between the tutor and the students and also the communication between the students who have the 
opportunity to create, share and exchange information and ideas in virtual communities and networks. Then, 
we present the evaluation study conducted and the results gathered regarding the tutors and students 
experiences, their thoughts and opinions. 

The structure of the rest of the article is as follows: In next Section, background topics on social media 
and web 2.0 are discussed and related works are presented. In Section 3, we describe the content and the 
methodologies of the online course developed, the topics it covers and the social technologies involved. In 
Section 4, the evaluation study conducted is presented and the results gathered are discussed. Finally, Section 
5 concludes the article and draws directions for future work. 


2. BACKGROUND TOPICS AND RELATED WORK 

Web 2.0 technologies and tools have been used with great success in the e-Learning field and can be utilized 
to improve the efficiency of the learning for the best benefit of the students. There are many definitions 
regarding the term and the concepts of Web 2.0 which is defined as the social use of the Web which allows 
people to collaborate, to get actively involved in creating content, to generate knowledge and to share 
information online (Grosseck et al 2009). Over the past few years, the integration of social media and web 
2.0 technologies in educational systems has attracted a lot of interest. A recent overview of good and widely 
used approaches can be found in (Moran et al 2011) (Tess 2013). Several studies (Venkateshet al 2014) 
(Churchill 2009) (Laird 2014) point out the important role that web 2.0 technologies can play in educational 
systems and in the e-learning area in order to assist tutors to enhance the efficiency of their courses' learning 
activities, engage and motivate students and improve the communication and cooperation among students 
and their tutors. Various studies (Holotescu 2015) (Huang et al 2014) examine the way that web 2.0 can 
assist students in learning and support the development of their skills and competencies and also enable drive 
broader pedagogical approaches and decision making with the aim to improve educational material and 
learning procedures. In the work presented in (Ebner et al 2010), authors present a research study that was 
carried in an Austrian University regarding the use of microblogging platforms in Higher Education. The 
authors indicate that microblogging should be seen and handled as a completely new form of communication 
that can support informal learning beyond the classrooms. Authors in (Hung et al. 2010) explore how social 
networking technologies can be used to supplement face to face courses and provide a mean of enhancing the 
students’ sense of community and promote classroom communities of practice in the context of higher 
education. The authors indicate that the majority of the participants in their experimental study developed a 
strong feeling of social connectedness and expressed favorable feelings regarding their learning experiences 
in the courses where social networks were used as a supporting tool. A study made by Callaghan (Callaghan 
et al 2012) shows the potential of social network integration into course curriculum. The authors point the 
critical role of the teacher in engaging effective online learning in social networks and indicate that the 
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quality of teacher-student relationships, the extent to which a ‘learning’ rather than ‘social’ attitude was 
established, and the online presence that the teacher exerted in the social network facilitate more successful 
student learning. In the study presented in (Arnold et al 2010), authors indicate that social networks can 
extend the community originally formed in a physical classroom into an online social network community. 
The online environment could provide students and teachers the ability to publicly post and read each other's 
work, modeling approaches to the assignments and various opportunities to provide feedback. Authors in the 
work presented in (Roblyer et al 2010) study the integration and the utilization of the Facebook platform in 
the education field. Authors present how students could communicate and cooperate formulating virtual 
classes and indicate that students are open to the possibility of using Facebook and similar technologies to 
support classwork. Finally, authors in (Grosseck et al 2009) examine the potential of the Twitter network in 
the educational field. The authors describe its capabilities and present various ways it can be utilized by the 
tutors in their courses. The work indicates that Twitter can be very useful and assistive for both tutors’ 
teaching and students’ learning enhancing their communication and cooperation. Researches and educators 
indicate the important role that social media can play in the educational field for assisting the tutors in 
teaching and students in learning for the best benefit of their learning progress. It is well pointed that teachers 
should have a strong knowledge and understanding of the way that social networks function, what each social 
network offers and how they can be utilized and integrated into the courses they teach. 


3. WEB 2.0 TECHNOLOGIES AND USING SOCIAL MEDIA 

In this section we present the educational system developed and the courses designed to assist tutors in 
learning how to use social media and web 2.0 technologies in education. The educational system offers tutors 
various functionalities. Initially, tutors register to the system. During registration, personal information such 
as name, age, sex, years of previous employment, discipline, school and specialty are required. After 
subscription the tutor can, at any time, enter the system and assess his account and then select and study the 
available courses. 

The system provides tutors a series of online courses that assist them to learn how to use web 2.0 
technologies and social media in education and also assist them to develop online courses and enhance their 
teaching. In general, the platform provides tutors with courses covering the utilization of the new generation 
technologies in education. The courses are organized into categories and each category consists of a series of 
courses. Three main categories of courses are available covering the domain of application of social media in 
education, presentation of web 2.0 technologies and the e-Learning practices in classroom and in distance 
education. A part of the categories of the online courses provided by the platform is illustrated in Figure 1 . 



Mobile training guidelines courses 
Web 2.0-based Mobile Technology in VET 


This course is also available in Bulgarian I HeTete na 6i/irapcKH 


Applying Social Media in VET 



Figure 1. A part of the main categories of the courses 
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Each category consists of a number of courses. A basic course regarding the use of social media in 
education is the “Applying Social Media in VET”. This course after an initial introduction to social media 
provides detailed, step-by-step instructions to use some of the most representative social media tools for 
education. The social media tools covered are Wikispaces, PrimaryPad, Twitter, TodaysMeet, Facebook, 
Edmondo and Pinterest. Some of these are very popular tools that the educators may already be familiar with 
(Facebook, Twitter) while others are not popular, but are very useful and easy to learn tools (PrimaryPad, 
TodaysMeet) that can be used in the classroom or in distance education. Each presentation/chapter has an 
introduction and instructions of how to use the tool and also a section with ideas, guidelines and scenarios to 
follow for the tool presented. The contents of the course are illustrated in Figure 2. 


applying Social Media in vet 


Home □ My courses □ MobiVET2.0 □ Applying Social Media in VET 
News forum 

Introduction 

0 Introduction 

Representative Tools 

0 Representative Tools 

Collaborative Projects 

0 Set up a Collaborative Site/Activity (Wikispaces) 

0 Simple collaborative writing (PrimaryPad) 

Blogs and Microblogs 

0 Create a Twitter account of your Course 

0 Create a simple back channel (TodaysMeet) 

Social Networking 

flt] Use a Facebook Page or Croup for your Course 

0 Create a Social Classroom (Edmondo) 

Content Communities 

ktl organize and share your resources in visual boards (Pinterest) 

Figure 2. Contents of the course “Applying Social Media in VET” 

The course provides an introduction to the social media and to the technologies it involves. It also 
provides a simple categorization of the basic types of Social Media Tools. Furthermore, it describes the 
benefits for both students and educators of applying social media in education and especially in distant 
learning. For each type of Social media tools (Collaborative, Blogs and Microblogs, Content Communities, 
Social Networking, Virtual Social Worlds, Virtual Game Worlds) it refers to specific tools that are popular or 
useful for education. For each tool there is a small introduction about what it is and what kind of functions it 
offers. It focuses on free and open source software that is either popular, so educators and students are 
probably already familiar with, or simple and easy to use in classroom. The platform also provides the 
capability for the trainers to create an online course using the Web2.0 technologies. It provides guidelines for 
designing and developing an e-Learning course for instructional designers and trainers who are new to the 
development of online learning resources and it addresses the design, development and delivery activities 
which are specific to e-Learning. The tutors can access the educational system anytime, select the courses 
they would like to take and study the material of each course on their own pace. Finally, while studying the 
courses, the tutors can utilize the tools they study into their own courses and help their students to explore 
and use them. 
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4. EVALUATION 

The evaluation study had the purpose to evaluate the teachers’ overview about the utility of didactical aspects 
of the system and the courses designed. It aims to provide an insight of tutors’ feelings regarding the 
utilization of web 2.0 technologies into their courses. The evaluation was conducted during the school year 
2014/15 and the participants were 11 tutors from Greek schools from different categories of schools and 
education levels. 

The primary descriptive analysis indicates that the respondents are distributed by gender with ratios of 
27% male and 73% female respectively. Also, regarding the respondents’ age, it can be observed that 73% of 
teachers are over 40 years old as illustrated in Figure 3. 


■ a) male ib) female 



0% 9% 

■ a) less than 25 years old 

■ b) 25 -30 years old 

■ c) 30-40 years old 

■ d) older than 40 



Figure 3. Tutors gender and age. 


In addition, the distribution of the educational levels of the tutors’ organizations is illustrated in Figure 4. 
The results showed that most of the tutors were teaching in secondary or vocational schools. 


o% 



■ a) secondary school 

n b) secondary grammar school 
(gymnazium) 

■ c) secondary vocational school 


■ d) post secondary (Higher 
Secondary / Junior College/ 
Church Post Secondary) 

■ e) Vocational School (ITS / 
MCAST) 

■ f) other type of secondary 
school 


Figure 4. Distribution of the educational levels of organizations 

The system provides online courses that the trainers studied and used into their courses. Presented in 
Figure 5 are the courses that tutors studied and utilized and the most studied courses were the “Applying 
Social Media” and “Web 2.0 based Mobile Technologies”. Both of these courses present technologies and 
tools that could be useful for all trainers regardless of their domain. Most of the other courses offer good 
practices regarding specific domains and as expected fewer trainers were inclined to use them. 
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d) E-Learning Practices in VET 

c) Applying Social Media in VET 
b) Web 2.0-based Mobile 
Technology in VET 
a) The Green Office 


0% 10% 20% 30% 40% 50% 

Figure 5. Courses attended by the tutors 

After tutors had studied the courses, they were asked to apply the practices and the learning activities they 
learned in their courses. They were also asked to integrate and use web 2.0 tools they find interesting and fit 
to the nature of their courses. Then, all trainers were asked to complete an online questionnaire which 
consisted of questions indicating their experiences of using social media in their lesson and evaluating the 
quality of the online courses. The questionnaire consisted of 10 questions. Questions 1-8 were based on a 
Likert (Likert 1932) scale (1: not at all/not good, 5: very much/very good) (see Table 1). Remaining 
questions 9-10 were of open type and concerned strong and weak points or problems using the platform. 


Table 1. Analysis of the tutors’ questionnaires 


Q 

Questions 

Answers (%) 

1 

2 

3 

4 

5 

l 

Did you find the online courses interesting? 

0 

9.1 

18.2 

18.2 

54.5 

2 

How much did the online courses assist you to utilize Web2.0 
Technologies into your courses? 

0 

0 

27.2 

36.4 

36.4 

3 

How much did the online courses assist you to use the social 
media in classroom? 

0 

18.2 

18.2 

18.2 

45.5 

4 

How would you rate the quality of the context of online 
courses? 

0 

0 

27.3 

18.2 

54.5 

5 

How would you rate the quality of the multimedia of online 
courses? 

0 

9.1 

18.2 

18.2 

54.5 

6 

How appropriate did you find the length of the 
course/courses? 

0 

0 

9.1 

45.5 

45.5 

7 

How would you rate the quality of the learning material in 
general? 

0 

0 

9.1 

45.5 

45.5 

8 

Would you suggest the online courses to other trainers? 

0 

0 

0 

45.5 

54.5 


The questionnaire results show that the tutors gave very positive responses. More specifically, tutors’ 
answers show that they found the system and the courses very helpful in learning how to utilize web 2.0 
technologies and social media in education. They also found the courses to be well structured and designed 
and the educational context to be of high quality. Finally, tutors indicate that they will definitely suggest the 
system and the courses to other tutors. 

For the second part of the evaluation study, tutors asked their students regarding their experience of using 
web 2.0 tools and social media in the courses. The purpose of the study was to examine the students’ 
thoughts regarding the efficiency and the effectiveness of web 2.0 tools and social media in the classroom 
and in distance learning. In this part of the study, the participants were 100 students. The fields of the studies 
that the tutors teach and students participated in were among others: theoretical studies in sciences as 
engineering. Physics, Economics and Business Administration and Mathematics. The questionnaire given to 
the students consisted of 5 questions which were based on a three point Likert-scale (strongly agree, neutral, 
disagree). The results collected are presented in Table 2. 
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Table 2. Analysis of the questionnaires that students filled in 




Agree/ 

Neutral 

Disagree 



strongly 


/ 


Questions 

agree 


strongly 

disagree 



% 

% 

% 

1 

The use of social media can enhance my learning interest. 

84 

12 

4 

2 

3 

The use of social media can enhance my learning motivation. 
The online course helped me learn more effectively. 

85 

70 

72 

10 

20 

22 

5 

10 

6 

4 

The social media helped me to better communicate and cooperate with 
my fellow students. 

75 

20 

5 

5 

Would you suggest the web 2.0 tools to be used by the next year’s class? 

82 

17 

1 


The results indicate that the students enjoyed and had a positive feeling regarding the use of web 2.0 tools 
and social media in education. It is very encouraging that a very high percentage of the students 
(approximately 85%) claimed that social media increased their learning interest and motivation. Students also 
indicated that the communication with their fellow students and their cooperation was better and that they 
were able to cooperate in a more direct way. Finally, most of the students (82%) would definitely suggest 
next year’s class to utilize the web 2.0 tools and the social media. 


5. CONCLUSION 

Social media and web 2.0 tools are important technologies that can enhance the learning capabilities of 
educational procedures both in classroom and in distance education. A vital aspect of the effective utilization 
of the social media and web 2.0 tools into courses and students’ training activities in both formal and 
informal learning is the tutor’s experiences and skills. However, research studies indicate that in many cases 
tutors are not familiar of the new web 2.0 technologies and are not aware of what they offer and how they can 
be utilized in education for the best benefit of the students’ learning. So, in this paper, we present an 
educational system and courses designed to assist tutors in learning how to utilize the capabilities offered by 
the Web 2.0 technologies and the social media and understand what each tool can offer and how they can be 
integrated in educational activities and in their courses in both classroom and distance education learning. 
The utilization of social media and Web 2.0 tools could enhance the interaction between the tutor and the 
students and also the communication between the students who have the opportunity to create, share, 
exchange and collaborate on information and ideas in virtual communities and networks. The evaluation study 
reports very encouraging results. Findings indicate that possible pitfalls of tutors or even students being 
unfamiliar with the Web 2.0 tools and technologies and the lack of institutional support can be easily 
overcome. 

As future work we plan to make a bigger scale evaluation of the educational system’s capabilities and 
furthermore extend the learning scenarios and the educational activities offered to the tutors during the 
designed courses. Furthermore, we plan to extend the system so that tutors that utilize web 2.0 tools into their 
courses, to be able to report and illustrate good practices and share their experiences and their students’ 
feelings with the educational community and with other tutors, stakeholders and institutions on local, national 
and international level. Exploring this direction is a key aspect of our future work. 
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ABSTRACT 

One of the most intriguing objectives when teaching computer science in mid-adolescence high school students is 
attracting and mainly maintaining their concentration within the limits of the class. A number of theories have been 
proposed and numerous methodologies have been applied, aiming to assist in the implementation of a personalized 
learning environment that approaches these objectives. In this paper, an empirical study is performed with the support of 
the “Hour of Code” initiative which took place for a second consecutive year at the end of previous year. The initiative 
was presented to a number of high-school students as a motivation for teaching computer programming to them. An 
evaluation of the students’ programming skills is attempted with the aid of a questionnaire and a simple personalization 
framework is presented in order to adapt to the students’ personal needs. 


KEYWORDS 

Teaching programming, evaluation. Hour of Code, personalization framework, adaptation 


1. INTRODUCTION 

School as a learning organization [9] but primarily as a social environment [1] and the educator as carrier and 
imparter of knowledge are daily facing many challenges in their attempt to shape the “school of the future”, 
where knowledge is not only statically presented but transferred as part of a dynamically and evolving 
procedure. Within such an environment computer science plays one of the most important roles. Computer 
science depicts not only as a static and tiresome lesson, but also as a continuously effort to present and 
transfer useful knowledge and moreover assist students in their quest for critical evaluation of the 
information presented. 

Teaching computer science to high school students clearly should be enlisted to Bruner’s pedagogical 
paradigm for constructivist learning approach where students are encouraged to enrich what they learn from 
the tutor with previous acquired knowledge from their past, in order to formulate new ideas or concepts [2], 
Combining this educational procedure with Piaget’s model for cognitive learning where students are taught 
something, think about it and then try to express it in their own words [8], lead to a hybrid teaching model for 
computer science in classroom. 

Moreover, computational thinking theory as introduced by S. Papert [7] and later presented by J. Wing 
[11], advocate that algorithmic thinking is a fundamental skill which everyone can realize (and not only 
computer scientists). In addition. J. Wing argued for the significance of bringing computational thinking in 
other aspects of life. 

Within this educational environment, bringing computer science and teaching programming lessons to 
high school students not only encourage their way of understanding through computational thinking and 
cognitive learning, but also promotes their personal skills and influence them in an alternative way of 
acquiring and using the knowledge learned. 
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All of these aspects described above are widely recognized and were included and applied in an 
innovative idea which took place worldwide during the 2nd week of December (8-14 Dec. 2014) which was 
named “Hour of Code” [12]. The “Hour Of Code” initiative firstly introduced in the United States of 
America, as an opportunity for mid-adolescence students to be taught “just one hour of code”. The campaign 
was quickly spread all over the world and moreover promoted from numerous powerful and influential 
people, including the president of the United States, founders of big IT companies and other individuals from 
sports and arts. The success of the initiative was significant and students throughout the world were taught 
and enjoyed an hour of code for a consecutive year. 

In the remainder of this paper, the “Hour of Code” initiative is briefly described in section 2 and the 
three-step teaching methodology which we applied in section 3. Furthermore a short evaluation of the 
students understanding of simple programming concepts with the aim of an additional example is given in 
section 4, along with a questionnaire filled by each student. Finally, we extend our previous work [6] by 
proposing a simple framework which aims to help us personalize the computer learning environment for each 
student and adapt our teaching methodology to their specific skills and needs. 


2. THE HOUR OF CODE 

2.1 Description of “The Hour of Code” Initiative 

The most significant objective of the “Hour of Code” initiative as this is presented by the originators of the 
idea is to bring closer all students to computer science and programming, through the presentation and 
accomplishment of the “one hour of code” paradigm. The idea initially originated in the United States of 
America from an non-profit organization (code.org) which then turned into a public charity. 

The significant importance of the initiative and the successful results which were obtained, are 
demonstrated by the numbers presented in the first page of their website. Namely, almost 5.2 billion lines of 
code have been written till today and more than 100 million students have learned at least one hour of code. 
The campaign of the initiative, as well as the programming examples which are presented, have been 
endorsed by great personalities of all the important aspects of social life. Moreover, even the president of the 
United States supported actively the project and encourages every student to actively participate and 
complete at least an hour of code because “...learning these skills isn’t just important for your future, it’s 
important for our country’s future. .”[10]. All these factors demonstrate the noteworthy significance of the 
initiative and exhibit many of the reasons which made the DG connect European Commission to organize a 
similar event within the boundaries of the European Union during the 3 rd week of October 2014 with 
excessive success. 


3. TEACHING METHODOLOGY 

3.1 Methodology of Teaching “Hour of Code” in the Classroom 

Within this context and the “Hour of Code” as the triggering event, we decided to actively participate in the 
initiative and teach the very first tutorial of the webpage to 3 different classes of high -school students aged 
14-17 years old. Most of the students had no previous experience in the field of computer programming (with 
the exception of a few of them who having been learning computer algorithms as part of a course in their 
curriculum) and mainly live in rural neighborhoods. 

The approach of teaching the computer programming hour is briefly explained in the 3 steps which follow 
and important characteristics are highlighted throughout this description. 

3.1.1 Introduction - Explaining the Importance 

The initial teaching approach which was selected in order to evenly introduce the importance of computer 
science (and more generally computers) to the students, was a 3 -minute brief introduction on the recent 
innovations on the computer science field (tablets, touch-screens, mobile devices) and the numerous 
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applications which nowadays all of us are daily using. In addition the students were encouraged to imagine 
their parents’ life 30 years ago in the decade of 80’s when the presence and use of computers were 
insignificant. 

3.1.2 Career Opportunities in the Computer Science Field 

Following the initial approach presented to the students for the context of computer science, another step 
which introduced career opportunities in the CS field was presented. A brief discussion in the form of 
questions-answers took place with the students showing a lot of curiosity for the working conditions and 
mainly the annual salary of professions relating to computers. A very short demonstration was made with 
pictures taken from the web which illustrated the new offices of Apple Inc., Google and Facebook. Students 
were excited to learn the middle IT computer annual salaries and the benefits which arise when working for a 
big IT company. 

At this point a short video from the “Hour of Code” webpage was showed, where a number of 
personalities analyzed the importance of computer science and programming. Students were impressed to 
learn that a considerable number of the top- 100 wealthiest people on earth are working in the IT industry. 

3.1.3 Demonstrating the 20 Steps Example 

The most important step of the teaching methodology followed was the demonstration to each class of the 
first example of the webpage with 20 steps. This example was carefully chosen from the organizers of the 
initiative to include the basic steps of programming, conditions (if- statements) and loops (repeat and 
repeat-until statements). 

Students enjoyed the demonstration (in a single PC with the image projected to a board) and the main 
reasons for this success were the simplicity of the examples (cartoon characters were chosen for the 20 steps) 
and the built-in video explanations of each group of paradigms from well -recognized celebrities. 

After the first few steps, each of the three classes was able to solve the additional steps roughly effortless 
and most of the students found the solutions to each step relatively quickly. The very last steps of the 
examples were more difficult and a few number of the students lost their attention throughout the whole hour. 


4. EVALUATION OF THE INITIATIVE FROM THE STUDENTS 

In order to evaluate the student’s perceptive of the programming patterns presented and their basic 
programming skills and potential programming talents, we performed a two phase evaluation approach. 

Initially, another example (Lightbot) of the initiative was given for each student to complete alone 
(without the educator’s assistance) during another teaching hour of the Computer Science course (one pc per 
student). 

The purpose of this phase was for the educator to quietly evaluate the student’s progress in the puzzle, for 
the proper usage of condition statements and loops. The example was more difficult than the one presented 
and a significant amount of the students did not accomplish it (till the end). All these results and the overall 
progress of each student separately, were recorded in an independently designed evaluation sheet. 

For the second phase of the evaluation, a specifically designed questionnaire was given to each student (in 
all 3 classes) with a few questions. The purpose of the questionnaire was for the student to evaluate the 
initiative, record their experiences in programming and their interest in computer science. Among the 
questions, one was specifically chosen to ask for their opinion of the initiative and give an estimate of their 
level of understanding of the programming patterns presented (all of them, many, few and not-at-all). The 
results obtained from this question are presented in Table I. 
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Table 1. Level of understanding (according to students opinion) 


Understanding of 
each step 

Number of students (49) 


1st class 
(age 15) 

2nd class 
(age 16) 

3nd 

class 

(age 

17) 

Percentage 

All 

5 

6 

3 

28,57% 

Many of them 

13 

6 

7 

53,06% 

Few of them 

4 


4 

16,33% 

None 

1 



2,04% 


Another noteworthy fact which was inferred from the questionnaire given was the wide acceptance of 
computer science among the students and their motivation for a future career in the IT field. Almost 3 out of 
4 students envision a profession in the computer science field (according to their understanding of computer 
science). Among them, there subsist a few which yearn for the hardware industry and few of them in the 
computer programming field (Table II below). 


Table 2. Possible future career in the IT industry 


Career in the IT 
field 

Number of students (49) 


1st class 
(age 15) 

2nd class 
(age 16) 

3nd class 
(age 17) 

Percentage 

Yes 

12 

9 

6 

55,1% 

No 

11 

3 

8 

44,9% 


5. PERSONALIZING THE COURSE - ADAPTING TO EACH 
INDIVIDUAL CLASSROOM REQUIREMENTS 

The assessment performed and the results obtained from both the evaluation sheet and the questionnaire 
given, motivated as to start designing a personalized system in order to improve the quality of teaching 
programming to these students and adapt our methods to their specific needs. 

Our goal was to personalize the learning environment for each individual and provide personalized 
material and exercises for the course of computer studies in high school. 

The necessity for a personalized education environment has been pointed out in numerous cases both in 
the past [4] as well as nowadays [3], [5]. The principal advantage for a personalized approach from a 
student’s perceptive is the ability to create an adapted learning environment with personalized notes and 
exercises which will guide the student to learn at its’ own pace. Therefore, our approach was to firstly design 
and in a later phase implement, a simple personalized learning environment in order to provide our students 
an enhanced learning experience. The implementation of our system is still in a premature phase; 
consequently we present here snippets of the initial design. 

In the first phase of our design of this personalized teaching environment, a simple user profile was 
created for each student. The profile comprised of basic characteristics for each student (name, age, gender) 
and two other fields (scienceedu and understand). Both of these fields were given with a decimal numeric 
score (0 to 1 ) which was obtained from the evaluation phase earlier described. The sciencedu field illustrate 
the student’s tendency to math/science and the understand field our estimate evaluation of the student’s 
understanding of programming. 

In the next phase, we are planning to create personalized material and small tests for three categories of 
computer programming. The material should be carefully chosen to illustrate specific components of 
programming, for three categories (beginners, intermediate and advanced) and adapt to each student’s 
requirements. 
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The main idea of this design is to support each student by assessing his/her skills and automatically 
placing them in one of the three categories. This classification will be based on the evaluation results 
obtained by the given questionnaire combined with the user profile’s numerical fields. 

Finally, a simple automated system will be designed and implemented in order to generate and present 
personalized material according to the age/gender of the student and ranking of the student. The proposed 
personalized system could initially perform a basic preliminary test for each student. The results obtained 
could identify potential weakness and errors that a student has and automatically generate the appropriate 
material which a student should study. 

This simple framework is in the process of design and implementation and any further results obtained 
should be tested in an experimental environment within the classroom. 


6. CONCLUSIONS AND FUTURE WORK 

Evaluating teaching performance and students’ learning behavior and skills are very challenging tasks which 
require effort and time. The opportunity to teach and assess students’ programming skills was given with the 
“Hour of Code” initiative. In this paper, a brief description of the initiative was given, along with the 
teaching methodology which was followed. 

The evaluation of the students’ learning behavior and the interest they expressed, motivated us with the 
idea to adapt our methodology and teach computer programming according to their specific needs. Therefore, 
a simple personalized framework is under design and implementation which aim to provide a personalized 
learning environment for the computer programming education. 

The work presented is an ongoing project with a number of open questions and challenging ideas. The 
framework presented just finished the implementation phase and a number of ongoing questions acquired our 
attention. The design is simple, but will provide us with enough material to teach for the remainder of the 
semester. 

Therefore, a new framework will encapsulate all those concepts and a design of an adaptive and 
personalized learning management system will assist instructors of algorithmic and programming courses. 
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ABSTRACT 

Usually, distribution electricians are called upon to solve technical problems found in electrical substations. In this 
project, we apply problem-based learning to a training program for electricians, with the help of a virtual reality 
environment that simulates a real substation. Using this virtual substation, users may safely practice maneuvers with 
varying degrees of complexity. To improve the user’s sense of immersion, interactive devices such as the Oculus Rift 
virtual reality headset are going to be adopted. The project’s stakeholders had a good impression of the device and 
believe that the proposed methodology will improve the training and effect positively the electricians’ performance, 
reducing accidents inside the substation area and decreasing the time required to reestablish the power supply after a 
failure. 
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1. INTRODUCTION 

The electric power system consists of power generation, transmission and distribution networks, and load 
consumption. The power generation is the origin of the electrical energy. A transmission network is 
composed by high-voltage power lines, which drive electric power from a generation station to a load center 
area, as they are usually far apart for long distances. A distribution system has medium and low voltage 
networks and conducts electrical energy from the transmission system to industrial, commercial and 
residential loads. These sections of the electric power system are linked by electric substations. Their 
function is to reduce or increase voltage levels in order to supply correctly each section of the system, 
thereby establishing a connection between the power generation and the transmission system on the one 
hand, and the transmission and distribution systems on the other (Massachusetts Institute of Technology, 
2011 ). 

A substation has the electrical and protection equipment necessary to transform power with safety and 
efficiency and offer an adequate power quality to the customers of the distribution system. A substation 
comprises transformers, regulators, circuit breakers, disconnectors, monitoring equipment, protection relays, 
electrical buses, and others (Blume, 2007). Since this facility is a fundamental section of the electrical 
system, it is very important to maintain its proper operation. Thus, if there is a problem in the substation after 
an electrical fault in its area, the site must be repaired as soon as possible. As a result, the customers would 
not have their energy supply interrupted for a long period and damages to the power grid or equipment of 
customers could be avoided. Electricians allowed to work in substations are the professionals who will 
respond in most events of technical problems in distribution substations in a local electric utility. 

In order to obtain the best performance of the electricians, electric companies must offer an efficient 
training to guide them properly. Therefore, the project in this paper proposes an improvement of electricians’ 
training program by including a problem-based learning methodology within an application of virtual reality 
(VR) in the current training program of the utility. A virtual substation, as much realistic as possible, with all 
its equipment has been developed to give electricians the opportunity to safely operate the equipment and 
execute basic and emergence maneuvers. 
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2. RELATED WORK 

2.1 Immersive Virtual Reality in Education and Training 

Adopting a VR environment for education and training can be very attractive for several reasons. First, as it 
does not involve security risks that may exist in the real world, users can freely explore the environment 
without harming themselves, their colleagues or even damaging the environment. Second, users can try to 
perform tasks and solve the proposed problems how many times they require, without the pressure of a 
classroom full of colleagues (Patel et al, 2006) or even a formal evaluation. Thus, this kind of environment 
encourages the user to conduct his/her learning at his/her own pace, with no risks (Mikropoulos et al, 1997). 
Given these characteristics, VR environments have been developed for different trainings (Grabowski and 
Jankowski, 2015; Sacks et al, 2013; Ausburn and Ausburn, 2004). 

A VR environment is not only capable of representing and simulating real or fictional environments, but 
also making users feel like they are physically present in those places (Ausburn and Ausburn, 2004). In order 
to be more effective and let the users focus only on the tasks that must be performed, the VR environment 
must be immersive. Usually, immersion can be achieved by covering the whole field of vision of the user 
with a Head Mounted Display (HMD). 

2.2 Electrical Simulators 

Electrical simulators are useful to help professionals to understand ordinary and emergence operations that 
he/she will face in a work daily basis. One example of electrical simulator is STOP (Silva, 2011). In STOP, 
users interact with single-line diagrams of electrical systems, simulating faults in the systems and setting 
relays, circuit breakers, transformers and other equipment. However, as the only way to interact with the 
systems is through the diagrams, users do not have a realistic experience of manipulating the equipment. 

Other simulators offer a more realistic experience, such as Virtual Substation and Furnas 02 (Silva, 2012). 
Both simulators aims to train new technicians to control and/or operate an electrical substation or a power 
plant, so that the facilities were modeled in 3D in order to look like a real substation and power plant. In fact, 
it is possible to interact with most of the equipment through joysticks and VR glasses, but the look and feel of 
these simulators is not as realistic as the project presented in this paper demands. 

It is not common to find substation simulators that use immersive VR to give the users a good impression 
of how a real substation is and. more important, how to operate the real equipment in a maintenance or 
emergency maneuver. A virtual electric substation does not replace a real one, although it is a valuable 
resource for training purposes, as a real substation may not be available for trainings. Even if it is available, 
several actions that are commonly performed in electric substations involves risks to the electricians, who 
may suffer injuries if performing unsecured maneuvers or lacking personal protective equipment, and to 
customers, who may face a blackout if a maneuver is wrongly done by the electricians. 


3. MATERIALS AND METHODS 

Nowadays, the training program of the electrical utility consists of theoretical and practical lessons. 
However, these practical lessons allow electricians to have only a very restricted contact with real substations 
so that it is nearly impossible to practice maneuvers during the training program. Therefore, a virtual 
substation designed as a VR environment will be a valuable tool to complement and improve the training 
program. Besides that, as the emergency calls in the electrical substation do not occur very often, the virtual 
substation will allow electricians to periodically practice and keep their skills sharp and proficient. 

To deliver the required realism, a real substation should be represented in the VR environment, containing 
accurate 3D models of all equipment and simulating the exact behavior of them. Moreover, as immersion is a 
must-have, HMD support is also required. To fulfill these requirements, the Unreal Engine (Epic Games, 
2015) was chosen, as it can render high quality graphics, is compatible with HMD and has a very active and 
supportive community. In addition, the selected HMD for this project was Oculus Rift (Oculus, 2015), as it is 
one of the most promising technologies available at the time that this paper was written, with a reasonable 
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support and rich online documentation. As the HMD covers the user’s field of vision, using a keyboard and a 
mouse could be a bit clumsy. Gesture-based user interfaces could provide a more natural interaction with the 
VR environment and are still under analysis by the authors. 

The behavior of the virtual substation was modeled by discrete event simulation, as its operations can be 
represented as a defined sequence of processes. Then, this virtual substation can be seen as a learning 
environment where instructors will select a specific issue that is avoiding the substation to work properly 
whereas students will have to investigate and isolate or solve that problem to fix the substation. In other 
words, students can learn by themselves in the environment while practicing to solve problems that may 
occur on real substations, as the Problem-Based Learning pedagogy recommends (Hmelo-Silver, 2004). This 
kind of methodology is characterized as an active learning, which was chosen because it is more attractive 
and effective for experienced professionals (Bonwell and Eison, 1991). 

To evaluate the performance of the student while restoring the substation, some criteria will be 
considered: proper usage of the personal protective equipment, completed tasks, correct interaction with the 
equipment to find the problem and a solution. Additionally, this assessment will be presented to instructors 
and students as a final score, similar to video games, enhancing the experience of the users and their 
motivations. 


4. PRELIMINARY RESULTS 

Although this is a work in progress project, it does have a few valuable results. At the time that this paper 
was written, about half of the 3D models of the equipment were designed and integrated into the VR 
environment. Figure 1 shows some of the models already included in the VR environment. 



Figure 1. 3D models for low voltage circuit breaker and disconnector (left) and details of an existing panel (right) 


To validate the behavior of the substation and the discrete event simulation, an emergency maneuver was 
implemented in the VR environment. In this emergency maneuver, part of the substation was turned off due 
to a short circuit, which means that customers were facing a blackout. Then, an electrician in the substation 
should identify the root cause of the short circuit, isolate it, and try to turn the substation on again - and all of 
these tasks should be performed by interacting with the substation’s equipment. To manipulate the 
equipment, the user can work with the Oculus Rift and a joystick or an ordinary monitor, keyboard and 
mouse. 

An informal evaluation of the VR environment and the emergency maneuver selected was done with 
technicians and instructors from the local electric utility. As there were some 3D placeholders in the 
environment instead of models of the real equipment, some stakeholders were not able to envision the 
substation correctly, indicating that the placeholders are not easy to understand. Therefore, the design of the 
3D models was re-prioritized and became a high priority task for the development and graphics teams. 
Despite this issue, the stakeholders were able to point a few pros of using the proposed VR environment: 
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• Improved electrician qualification, resulting on improved performance when some maneuvers 
are required in a substation ; 

• Reduced time to re-establish energy after a blackout, given that electricians will be able to 
successfully conclude more emergency maneuvers in a shorter time than nowadays; 

• Reduced accident rate in substations, as electricians will be able to practice good safety practices 
when performing the maneuvers in the VR environment. 

The pros indicate the potential of this project to benefit both the electricians and the society as a whole 
since the substation issues that cause blackout may be quickly repaired due to a better qualification. 


5. CONCLUSION 

This paper presented the project of a virtual substation being developed as an immersive VR environment, 
which offers a safe training area and represents accurately the behavior and operation of the system modeled. 
The proposed environment should at least partially suppress the current lack of practice of electricians in real 
substations, enabling them to explore the virtual substation and practice basic and emergency maneuvers, 
applying all concepts learned in theory classes with safety and without external pressure. An improvement on 
the electricians’ performance is also expected, in the form of reduced accident rates and shorter time to re- 
establish the energy supply. 

In the informal evaluation with the stakeholders, it was noticed that they expected a 3D scenario designed 
before the beginning of the maneuvers’ simulation, exposing that having a realistic environment can call 
more attention than a simulation case without external features. Nevertheless, they were positive with the 
project’s progress and with the immersive sense given by the use of the Oculus Rift. Other interactive devices 
(for instance, those with motion detection) are under analysis to improve the experience in the VR 
environment. 
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ABSTRACT 

The study adopted an Ex-post facto research design. Randomization sampling technique was used to select 346 
undergraduate distance learners and the learners were grouped into four. High and Low Goal setter learners and High and 
Low Decision-making skills learners. The instruments for data collection were Undergraduate Academic Goal Setting 
Scale (UAGSS), Undergraduate Decision-making Skills Scale (UADMSS) and Semester Examination Course Result 
Grades in four courses. The instruments were validated with face and content validity and a test re-test administered after 
three weeks. The obtained alpha value were 0.77 and 0.84 while, Semester Examination Course Result Grades for the 
four courses that formed the academic performance (Use of English and Communication skills 1, Use of English and 
Communication skills 11, History and Philosophy of Science and the Good Study Guides) were moderated by External 
examiners appointed by the Senate of the University, t - test for unequal samples and Pearson Moment Correlation 
Coefficient statistical method were used to analyse the data. The findings indicated statistically significant difference 
between high and low goal setter undergraduate distance learners academic performance (t = 4.01 < 0.000), statistically 
significant difference between high and low decision - making skills undergraduate distance learners academic 
performance (t = -3.35.93 < 0.001), statistically significant positive relationship between high goal setter undergraduate 
distance learners and high decision-making skills undergraduate distance learners academic performance (0.297<0.000) 
and statistically significant negative relationship between low goal setter undergraduate distance learners and low 
decision-making skills undergraduate distance learners academic performance (-0.307< 0.000). Recommendations were 
made towards learners’ retention and provision of support services. 


KEYWORDS 

Goal setting. Decision-making skills, Academic performance. Retention and Support services. 


1. INTRODUCTION 

Setting achievable goals are vital to human success and achievements in life. Executing a set of achievable 
goals will automatically depend on a foundation of taking effective decision towards actualization of the set 
goals. Since goals contribute to success, it must also influence students’ academic performance particularly in 
distance education. When learning goals are formulated by learners, it must be carried out with efficient 
decision-making skills (Aminu &Gali, 2012). Education and Counselling psychologists believed that learning 
to formulate goals and developing efficient decision-making skills are some of the basic training that should 
be inculcated in learners, if there must be successful academic records (Shertzer & Stone, 1976; Saka, 2006; 
Sambo, 2008 and Aminu & Gali, 2012). Setting attainable goals, determines the success of any activity, 
engagement or commitment. When goals are clearly mapped out, it must be accompanied with efficient 
decision-making skills so as to ensure it achievable implementation (Mind tools, 2014). Students’ academic 
performance in learning tasks and exercises like any other worthwhile activities must therefore be planned 
and properly guided. Distance learners all over the world are characterized by different engagements and 
commitments at a given time, as most of them combine work and study together (Ipaye,2003). Assisting 
distance learners set achievable goals as well as develop efficient decision-making skills is crucial toward 
meeting up with the increasing demand for distance related academic programmes all over the world. 
Distance learners need to work towards the achievement of these formulated goals regularly at the beginning 
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of the programme, during and at the end of the programme, if the learner wants to get the most out of them. 
Learners that set realistic goals can be more motivated than those learners that set unachievable goals (Ipaye, 
2003). Setting realistic goals have been traced to effective motivation. For example, Locke & Latham’s 
theory, 1960’s (cited in Mind Tools, 2014) showed that clear goals and appropriate feedback motivate 
individual. For undergraduate distance learners in open and distance education to set realistic goals, Locke 
and Latham (1990) outlined five principles of goal setting that need to be examined. The five principles 
outlined by Locke and Latham (1990) that were researched to improve individual chances of success are: 
Clarity, Challenge, Commitment, Feedback and task complexity. The researchers’ view was that, the 
application of these principles to learners will improve academic performance. Undergraduate distance 
learners in open and distance education and institutions may find these principles difficult to understand 
comprehend or apply due to their numerous engagements combined with learning at a distance. Apart from 
having a set of realistic goals as a distance learner, the other related and intervening variable is the learner’s 
decision-making skills (Aminu &Gali, 2012). Learner decision-making skills are vital in carrying out set 
goals. For distance learners to develop effective decision-making skills needed for goal setting, the learner 
must learn to take the following stages of decision-making: Efficient evaluation of problems, list of possible 
solutions in term of merits and demerits, application of the most appropriate solution, acceptance of the 
outcomes and acting upon the outcomes (Shertzer & Stone, 1976; Okon, 1983; Denger, 1986; Saka, 2006; 
Sambo, 2008; Aminu & Gali, 2012). According to Mind Tools (2014) stages involved in making efficient 
decision are: Establishing a positive decision-making environment, generating potential solutions. Evaluating 
alternatives. Deciding, Checking the decision, Communicating and Implementing. 

Assisting learners to effectively set realistic and achievable goals and developing in the learner efficient 
decision-making skills is one of the roles of distance learning institutions, particularly in the National Open 
University of Nigeria (Ipaye, 2003). The convention that learners must set goals and make rational decisions 
that will enhance their academic performance have generated different but related studies (Tanglang & 
Aminu, 2014;Joo, Bong & Choi 2000; Chan, Yum, Ran, Jegede & Taplin 1999). In another study, Roblyer 
(1999) worked on factors that motivate community college and virtually high school students to choose 
online or traditional course formats. The findings indicated that for students who choose distance learning, 
control over face and timing of learning was more important; for students who choose face-to-face (FTF) 
course interaction with instructor and students was paramount. Cheners. Hu, & Garcia (2001), Pajares & 
Kranzler (2002), Mclsaac, & Gurawerdena (1996) discovers that self-efficacy was related both to academic 
performance (r=38) and to persistence (r=34). Michael (2013) examined the evidence for the effectiveness of 
active learning. It defines the common forms of active learning most relevant for engineering faculty and 
critically examined the core element of each method and found that there is broad but uneven support for the 
core elements of active, collaborative, cooperative and problem-based learning. Examining these findings, 
none have specifically investigated goal setting and decision-making skills in relation to academic 
performance of undergraduate distance learners. The need to assist the distance learners to set realistic and 
achievable goals and acquire appropriate decision-making skills related to the demand of distance learning 
and education becomes necessary. 

1.1 Statement of the Problem 

The negative trend in the completion rate of learners in distance education programme when compared with 
general enrolment prompted this research study, cited in Tanglang &Aminu (2014). Many of learners don’t 
realize the importance of setting goals that will guide their learning activities, learning style and other 
activities that they combined with learning before and during the programme. From the researchers 
experience while interacting with undergraduate distance learners, many of these learners who were able to 
set realistic and attainable goals still lack the appropriate decision-making skills to carry out the set goals. 
Based on this, goal setting, decision-making skills and academic performance of undergraduate distance 
learners are been investigated in an attempt to draw implications for distance learners retention and support 
services. 

1.2 Research Hypothesis 

The following null-hypothesis was formulated for the study: 
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i. There is no significant difference between the academic performances of High and low goal setter 
distance learners, ii. There is no significant difference between the academic performances of High 
and low decision-making skill distance learners, iii. There is no significant relationship between the 
academic performance of High goal setter and high decision-making skill distance learners, iv. 
There is no significant relationship between the academic performance of Low goal setter and low 
decision-making skill distance learners. 


2. METHODOLOGY 

2.1 Research Design 

Ex-post factor research design was adopted for the study. 

2.2 Population, Sample and Sampling Technique 

The population consisted of an estimated Three thousand (3,000) One hundred levels undergraduate distance 
learners of the National Open University of Nigeria Lagos Study Centre (NOUN: Academic office, 2014). 
Three hundred and forty-six (346) One hundred level undergraduate distance learners were randomly 
sampled. 

2.3 Data Collection Instruments 

Data collection instruments consisted of Undergraduate Academic Goal Setting Scale (UAGSS), 
Undergraduate Academic Decision-making Skills Scale (UADMSS) and End of semester examination course 
result grades in the four core compulsory courses. The Undergraduate Academic Goal Setting Scale and 
Undergraduate Academic Decision-making Skills Scale were modified scales designed by Mind Tools 
(2014), a web-based programme for training individuals on how to develop realistic goals and make effective 
/ good decisions. The modified UADMS scale was first used by Tanglang &Aminu (2014). The UAGS scale 
consisted of eighteen statements on a format of Not at all with score of 5, rarely with score of 4, sometimes 
with score of 3, often with score of 2 and Very often with score of 1. The UADMS scale also consisted of 
eighteen statements on a format of Not at all with score of 1 , Rarely with score of 2, Sometimes with score of 

3, Often with score of 4 and very often with score of 5 on which the samples responded by honestly ticking 
the type that best describe their levels of goal setting and decision-making skills. The UAGS scale followed 
the five principles (clarity, challenge, commitment, feedback and task complexity) required for improved 
personal goal setting (Lock and Latham in Mind Tools, 2014). The statements on the scale were designed to 
measure: A). Preparing to set goals B). Forming goals C). Motivation D). Achieving Goals. The UADMS 
scale on the other hand, measures the following skills that were theoretically believed to form good /effective 
decision-making skills of individuals. Shertzer and Stone (1976); Okon (2003); Denga (1986); Saka (2008); 
Sambo (2010); Aminu, and Gali (2012); Mind Tools (2014) and Tanglang and Aminu (2014): A) 
Establishing a Positive Decision-Making Environment B) Generating Potential Solutions, C) Evaluating 
Alternatives D) Deciding, E) Checking the Decision f) Communicating and Implementing. Undergraduate 
learner with score range of 18 to 66 were interpreted as Low goal setters and low decision- making skill 
learners, while undergraduate distance learners with score range of 67 to 90 were interpreted as High goal 
setters and high decision-making skill learners. The interpretations served as basis for grouping the samples 
with One hundred and fifty-five High goal setters. One hundred and ninety-one Low goal setters. One 
hundred and forty-seven High decision-making skills learners and One hundred and ninety-nine Low 
decision-making skill leaners. The mean age was 20.31. 
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2.4 Validation of the Research Instruments 

The Psychometric properties of UAGSS and UADMSS were established through face and content validity on 
a pilot study that involved 50 Undergraduate distance learners of the university’s study centre in Kano. A 
test re-test conducted after three weeks of the first administration gives coefficient alpha values of 0.712 for 
UAGSS and 0.831 for the UADMSS. For the individual statements that measure the processes in setting 
realistic and achievable goals, the following Cronbach alpha reliability estimates were obtained: A). 
Preparing to set goals = 0.79. B). Forming goals = 0.67. C). Motivation = 0.68. D). Achieving Goals = 0.63. 
Individual statements that measures the processes involved in effective decision making, the following 
Cronbach alpha reliability estimates were obtained: A) Establishing a positive decision-making environment 
= 0.77. B) Generating potential solutions = 0.71. C) Evaluating alternatives = 0.77, D) Deciding = 0.81, E) 
Checking the decision = 0.80, F) Communicating and implementing = 0.799. The alpha values for each item 
on the two scales indicated their suitability for the study. While, the validation of the 2013/2014 semester 
examination for the four core-compulsory courses of the academic performance were established by External 
Examiners in the subject areas appointed by the Senate of the National Open University of Nigeria. 

2.5 Administration and Collection of Research Instruments 

The researchers were personally involved in the administration of the research instruments and also collected 
and compiled the required academic performance course grade scores of the samples. 

2.6 Methods of Data Analysis 

t-test for unequal samples was used to analyse the significant differences between high and low goal setters 
and high and low decision-making skill learners’ academic performance while, Pearson Products Moment 
Correlation Coefficient method was used to analyse the significant relationships between High goal setters 
and high decision-making skills learners academic performance and also for the analysis of low goal setters 
and low decision-making skills learners academic performance. 

2.7 Results 

The results of the study are hereby presented in table 1 and 2. The interpretation of each result follows the 
tables: 

Table 1 contained the analysis of hypotheses 1 and 2 which states that: There is no significant difference 
between the academic performances of High and low goal setter distance learners and that; there is no 
significant difference between the academic performances of High and low decision-making skill distance 
learners. 

Table 1 . t - test Analysis of Significant Difference between the Academic Performances of High and Low goal setter 
Distance Learners and Significant Difference between the Academic Performances of High and Low Decision-making 

skill Distance Learners 


Variables 

N 

Mean 

SD 

t- value P- Value 

High Goal Setter Learners 

155 

8.40 

1.63 

4.01 0.000 

Low Goal Setter Learners 

191 

7.78 

1.02 


High Decision-making Skills 
Learners 

147 

7.71 

1.23 -3.35 

0.001 

Low Decision-making Skills 
Learners 

199 

3.20 

1.44 
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Table 1 shows significant differences between the academic performances of High and Low goal setter 
distance learners and also significant difference between the academic performances of High and Low 
decision-making skills distance learners. 

Table 2 contained hypothesis 3 and 4 which states that there is no significant relationship between the 
academic performances of High goal setters and high decision-making skill distance learners and that there is 
no significant relationship between the academic performances of Low goal setters and low decision-making 
skill distance learners. 

Table 2. Relationship between the Academic Performances of High Goal Setters and High Decision-making Skill 
distance learners and Relationship between the Academic Performances of Low Goal Setters and Low Decision-making 

Skill Distance Learners 


Variables 

N r -value 

Sig. (2-tailed) 

High Goal Setter Learners 

High Decision-making Skills Learners 

302 0.297 

0.000 

Low Goal Setter Learners 

Low Decision-making Skills Learners 

390 -0.307 

0.000 


Table 2 shows significant positive relationship between the academic performances of high goal setter 
distance leaners and high decision-making skill distance learners’ and also significant negative relationship 
between the academic performances of low goal setter s and low decision-making skill distance learners. 

2.8 Discussions 

The significant connection between goals setting and decision-making skills as displayed in the academic 
performance can be linked to what was alluded to by Tanglang &Aminu (2014) that learner decision-making 
skills are vital in carrying out stated goals. The findings supported Joo, Bong & Choi (2000) that student’s 
self-efficacy for self-regulated learning positively related to his / her academic self-efficacy, strategy use, and 
internet self-efficacy. Cheners, Hu &Garcia (2001) was also supported by these findings which showed that 
self-efficacy was related both to academic performance and to persistence. In a similar context, Pajares & 
Kranzler (2002) on the study of direct effect of mathematics self-efficacy on mathematics performance 
discovered strong effect of general mental ability, this shows an interplay of self-efficacy as one of the 
variables of goal setting and effective decision-making skills. Similarlly, the findings supported Mclsaac, & 
Gurawerdena (1996) that a combination of cognitive style, personality characteristics and self-expectations is 
asserted to be able to predict the achievement in distance education. 

2.9 Implications for Learners’ Retention and Support Services 

The unit for learner support services in open and distance learning institutions can be used to train distance 
learners on setting goals and developing effective decision-making skills. First, student counsellors must first 
prove to the learners, the roles played by setting achievable goals and how effective decision -making skills 
compliment the achievement of set goals and their influence on better academic performance. Second, 
student counsellors can map out the components involved in goal settings and decision-making skills based 
on their theoretical foundations. Third, the components can be developed into steps and provided in a module 
to be followed in developing the skills needed for appropriate goal settings and effective decision-making 
skills for learners. Fourth, the student counsellors can assist learners on how the module can be acquired to 
arrive at setting their various goals and decision-making. 


3. CONCLUSION 

The findings of this study proved the advantage of goal setting and decision-making skills on academic 
performance of undergraduate distance learners. The study was limited to undergraduate distance learners 
therefore, there is the need for study that will compare goal setting and academic performance of open and 
distance learners with that of conventional institutions. 
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ABSTRACT 

Mobile devices and the interactions that these technologies afford have the potential to change the face and nature of 
education in our schools. Indeed, mobile technological advances are seen to offer better access to educational material 
and new interactive ways to learn. However, the question arises, as to whether these new technologies are being used to 
their full ‘interactive’ potential? In Wales, the Welsh language strategy (2012-2017) is pushing to ensure that the Welsh 
language can embrace new technologies and use them to support and facilitate it’s use by Welsh language learners and 
speakers. As more and more Welsh schools invest in ipads and handheld devices, it is important that the way these 
schools use these technologies evolves too. These new technologies (and the new interactions that they provide) need to 
be closely stitched with the desired learning objectives to ensure value, new meaning and learner engagement is achieved. 
This paper reports on an initial empirical study into the design of a Welsh Learning application, particularly the 
alignment of the interaction facilities of the technology and learning material in an attempt to harness the desired learning 
experience. 


KEYWORDS 

Elearning, mobile technologies, Welsh language, engagement, interaction, mobile learning 


1. INTRODUCTION 

As more and more schools are investing in mobile technology for learning (i.e. iPads, iPods and other 
handheld devices etc.), the question arises as to what these devices and the new modes of interactivity that 
they bring to the classroom will actually mean for education. This paper is focused on the education of the 
Welsh language, as Wales is currently losing around 2,000-3,000 Welsh speakers every single year (Welsh 
Language Commissioner, 2012). In fact, ‘there are many shortcomings in the teaching and learning of Welsh 
as a second language’ (Lewis, 2012). In terms of technology, there are an ever increasing range of exciting 
and engaging educational websites and applications out there; however, it seems that there are very few that 
support the education of the Welsh language (Welsh Government, 2013). Even though many Welsh language 
schools are moving toward the use of mobile devices, they might not be using the full potential of these 
devices to fully nurture a deep learning of the language. The aim of this research is to explore how the iPad 
device can be harnessed to create an interactive, fun, yet meaningful way of learning the Welsh language. Of 
particular interest is the relationship between interaction and learning and how these need to be closely 
aligned and nurtured in the mobile environment to ensure an ‘engaging’ learning experience is achieved. 


2. TRANSFORMATION - AN ENGAGED’ LEARNING EXPERIENCE 

Tablets, mobiles and handheld computers such as the iPad / iPhone are fast becoming the epi -centre of 
people's lives; they are transforming the ways in which we interact with technology. Most of these smart 
devices are currently embedded with powerful interaction facilities and programmable sensors such as GPS, 
gyroscope, microphone, haptic sensors, cameras, accelerometers and so on. In fact, these sensor-enabled 
devices are influencing all aspects of our social ecosystem, particularly holding huge pedagogical potential 
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for our educational system (Serrano-Santoyo & Organista-Sandoval, 2010). With sensors and new modes of 
interactions, these devices are changing the way we interact with the physical world around us; they afford 
interactions which play an intrinsic role in the learning experience. Indeed, interaction has the potential to 
draw out the essential quality of a learning object; it can initiate an absorbed interplay of cognitive and non 
cognitive faculties that in turn can cause what Dewey (1934) describes as a transformation - an ‘engaged’ 
learning experience. As Elkind (2009) highlights ‘[learning] experience is not always independent of our 
behaviour but rather created by it’. With mobile technologies, learners and educators of today, do not only 
have the opportunity to explore and reinvent the learning object in new and refreshing ways, but they also 
have the ability to expand learning through the amplification of social conversations and learning networks. 

In an attempt to understand these new opportunities for interactivity and the opportunity for learning, it is 
equally important to be aware of the concepts inconclusively linked to these new technologies and 
interactions. Usability, being one of these, can be seen as being a quality attribute that assesses how easy user 
interfaces are to be used (Nielsen, 2003). From the beginning, the study of how people use technology has 
played a central role in the development of Human Computer Interactions (HCI), so much so, that the HCI 
discipline has often been portrayed ‘as the study and the practice of usability. It is about understanding and 
creating software and other technology that people will want to use, will be able to use and will find effective 
when used’ (Carroll. 2001, p. xxvii). However, in terms of interaction and particularly the varied interactions 
of today, it is now evident that we need to understand the bigger picture and acknowledge that usability is 
actually made up of several components. Indeed, usability is a collective term for all aspects of an activity’s 
performance. As Lowgren & Stolterman (2004, p.5) describe, the design of computer interactions is more 
‘the process that is arranged within existing resource constraints to create, shape, and decide all use-oriented 
qualities (structural, functional, ethical, and aesthetic) of a digital artefact for one or many clients’. It is the 
shaping of digital artefacts for better user experiences, in terms of both appearance and function; the 
interaction point is what brings everything together and where new connections of knowledge emerge that 
hold the potential for greater understanding and learning. 

How might we shape new interactions for Welsh language learning? A number of researchers in the field 
of Learning Technology have chosen to use a Design-Based Research approach (Kelly, 2004; Bannan- 
Ritland, 2003; Barab & Squire, 2004) to examine and analyze in a systematic way every aspect of a new 
learning design innovation. The transformation, we discuss, depends on balancing the interaction afforded by 
the technology (function and appearance) with the learning cues and connections to guide the learner into a 
sense of flow and engagement in the learning environment. Interactions, particularly those afforded by new 
mobile technologies (touch screen, sensors etc), have the potential to expose the learner to a number of 
opportunities for the creation and recreation of new, exciting and ‘engaging’ learning experiences. The key 
lies with the learning designer and how well the ‘transformation’ is designed; how well the interactions, 
function as well as appearance, are aligned with the learning material. 


3. INITIAL STUDY OF THE APPLICATION 

This research was undertaken at the University of South Wales to explore how the iPad device and the 
interactions it affords can enhance the ‘Welsh language’ learning experience. In Wales, there is a ‘need to 
ensure that the [Welsh] language can embrace new technologies and use them to support and facilitate its use 
by Welsh speakers of all ages’ (Welsh Government, 2011). To build the Welsh language learning 
application, a design-based research approach has been used to ensure that each stage of the development has 
been considered and addressed as described (Carroll and Kop, 201 1 ). From the start, it was important to think 
about the appearance and function of the interaction and how this integrates with the learning objectives - to 
create ideas of what the main features of the application might look like and also how these would function 
(i.e. what interactions they would afford). Based on the national curriculum of Wales, we decided to focus 
the Welsh language learning application on the area of ‘Knowledge and Understanding of the World’ and 
within this to concentrate on three main topics: ‘Knowledge of the Home’ ‘Animals’ and ‘Weather 
Observation’. For example, on paper, the home environment section of the application aimed to encourage 
the child’s Welsh literacy skills and to reinforce their vocabulary and knowledge of items around the house. 
This core idea was then expanded through interaction to incorporate more playful experiences. In detail, the 
learner is presented with the cutaway profile of a house showing four separate rooms full of everyday 
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household objects. Each room contains objects that have been embedded with touch gesture interactivity and 
it is the learner’s task to visually explore the rooms, touch the objects and in doing so build up his or her 
Welsh language skills. The key lies in the interaction - driven by the curious exploration - in that each 
learner needs to be enticed to visually engage with an object and then to physically touch it to trigger a 
response. For example, on touching the television set, the television switches on and the Welsh pronunciation 
of the word ‘television’ is audibly played as well as textually presented. In doing so, the learner is able to 
familiarize him or herself with the sound, spelling and visual representation of each object in Welsh. 

Five local welsh medium nursery/ primary schools (1-5) were approached in order to test the application. 
Both children (approximately ten learners per class/ school) and teaching staff (approximately one teacher 
and two teaching assistants per class/ school) were asked to take part in the study; the study was conducted 
through the medium of Welsh. Observational methods and focus groups were used to evaluate the children’s 
impressions of the Welsh Language Learning application. The teaching staff from each school were asked to 
complete a questionnaire after using and observing the use of the Welsh Language Learning application to 
evaluate its interactivity and potential for learning. 

The overall perception from the study was that all children enjoyed using the application, they were all 
interested and it proved a suitable tool to engage and keep them involved. Generally, all children participated 
in the activities and enjoyed discussing the application with their peers. The playfulness of the drag and drop 
weather activity and the novelty of the dressing up an avatar seemed to be most popular with schools 
(1)(2)(3) whilst children in schools (4)(5) engaged more in the exploration of the home environment section 
as they found identifying all the items around the house as well as the quiz the more challenging. Some 
children did slightly struggle with the learning content and children from school (3) were recorded to 
occasionally use the ‘English Help’ button for help. Overall, as the focus group discussion highlighted, all 
children enjoyed and were interested in using the application; in fact, they wanted to use it more. From the 
questionnaire data, we concluded that teachers from all five schools felt that the children enjoyed, were 
interested in and had the potential to benefit educationally from using the application. Three out of five 
teachers thought that the ‘weather’ section seemed most successful with the children whereas school (2) 
thought that the home environment section was the more engaging. All teachers agreed that the majority of 
children were participating as they usually do and were all paying attention (i.e. not easily getting distracted 
etc). In terms of learning the Welsh language, four out of five of the teachers found that the application was 
slightly too easy for the target audience (i.e. very few children seemed to need extra guidance). However, 
when asked to rate what they thought the probability of learning for each child after using this application 
would be (on a scale of 1-5, 1 being lowest - 5 being highest), teachers from four schools rated the 
application as a ‘4’ for its potential for learning. The results also indicate that ‘learning through play’, 
particularly the playful elements incorporated into the interactions, proved to be an effective way to engage 
children of such a young age in the ‘Welsh language’ learning process. In terms of interactivity, all teachers 
felt that the children enjoyed interacting with the application; in fact, they all felt the need for more 
interactions to further enhance the child’s learning experience. 


4. CONCLUSION 

This short study has allowed us to identify the learning activities and interactions that the children were most 
engaged in whilst using the Welsh Language Learning application. More importantly, it has given us some 
insight into why the children were engaged with them (i.e. novel activities, playful interactions and 
challenging tasks). It is this power to ‘engage’ learners that emphasizes the importance of interaction to the 
learning process. In fact, it is this very ‘power’ that exposes the need to rethink the potential of the 
interactions that new technologies afford to the learning experience. Indeed, it is the attention to both the 
visual appearance and function of the interaction that can keep participants in the flow of their activities and 
consequently enhances their experiences. It is this detailed focus on the interaction that then plays an 
important role in determining the ‘transformation’. As we have seen from this initial study, it was a 
combination of challenging learning tasks (i.e. appropriately set learning objectives) framed in novel 
activities that were brought to life through visually and functionally ‘playful’ interactions that harnessed the 
learners’ engagement and the enhancement of the learning process. Looking ahead, the aim is to explore 
further iterations of this application with a deeper focus on the visual and functional aspects of the mobile 
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interaction and its role in the learning experience. Engagement is the key and it is the harnessing of 
engagement which takes interaction beyond usability and more towards the experiential, introducing many 
possibilities for new and exciting learning experiences. 


REFERENCES 

Bannan-Ritland, B. (2003) The Role of Design in Research: The Integrative Learning Design Framework, Educational 
Researcher, 32, 21-24Carroll, F. & Kop, R. (2011). “A Leanring, Research and Development Framework to Design 
for a Holistic Learning Experience”, E-Learning and Digital Media Journal. ISSN 2042-7530. 

Barab, S. and Squire, K. (2004) Design-Based Research: Putting a Stake in the Ground, The Journal of the Learning 
Sciences, 13, 1, pg.1-14 

Carroll, J.M. (2001). Human Computer Interaction in the Millennium. New York: ACM Press, Addison -Wesley, p. Xxvii 

Carroll, F. & Kop, R. (2011). “A Learning, Research and Development Framework to Design for a Holistic Learning 
Experience”, E-Lxarning and Digital Media Journal. ISSN 2042-7530. 

Dewey, J. (1934). Art as Experience. London: The Berkley Publishing Group.p.22 

Elkind, D. (2009). Learning through Play. Available at: 

http://www.communityplaythings.com/resources/articles/2009/learning-through-play. ( 1 8/03/2015) 

Lowgren, J. & Stolterman, E. (2004). Thoughtful interaction Design - a Design Perspective on Information Technology. 
USA: MIT Press.p.5 

Kelly, A. (2004) Design Research in Education: Yes. but is it Methodological?The Journal of the Learning Sciences, 
13,1, 

Lewis, G. (2012). Welsh Second Language in the National Curriculum -the case of teaching Welsh in English-medium 
schools, p. 2. Available at: http://www.assemblywales.org/paper_3_-_dr_w_gwyn_lewis.pdf (Accessed: 10/7/ 2013) 

Nielsen, J. (2003). An introduction to usability. (Online). Available from: http://www.useit.com/alertbox/20030825.html 
(31/10/2006). 

Serrano-Santoyo, A., & Organista-Sandoval, J.( 2010 ). “Challenges and opportunities to support learning with mobile 
devices”; In Proceedings of the 3rd Mexican Workshop on Human Computer Interaction, Universidad Politecnica de 
San Luis Potosl,. Mexico, 85-87. 

Welsh Government (2011). Written Statement - Welsh Government’s Welsh Language Annual Report for 2010 / 11. 
Available at: http://wales.gov.uk/about/cabinet/cabinetstatements/201 l/09decemberwelshlanguagereport/?lang=en 
(10/7/2013) 

Welsh Language Commissioner (2012) Data / Statistics. Available at: 
http://www.Welshlanguagecommissioner.org/English/Assistance/Pages/datastatistics.aspx (10/7/ 2013) 

Welsh Government (2013). Welsh-language technology and digital media action plan. Available at: 
http://wales.gov. uk/topics/welshlanguage/publications/welsh-language-technology-and-digital-media/?lang=en 
(12/07/2013)' 


150 



International Conference e-Leaming 2015 


DIGITAL RESOURCE EXCHANGE ABOUT MUSIC 
(DREAM): PHASE 2 USABILITY TESTING 


Rena Upitis 1 , Karen Boese 1 , Philip C. Abrami 2 andZaeem Anwar 1 

1 Queen’s University , Kingston, ON Canada 
2 Concordia University, Montreal, QC, Canada 


ABSTRACT 

The Digital Resource Exchange About Music (DREAM) is a virtual space for exchanging information about digital 
learning tools. The purpose of the present study was to determine how users responded to DREAM in the first four 
months after its public release. This study is the second phase of usability research on DREAM, and was conducted to 
guide future development of the tool. Research questions were designed to determine patterns of use for DREAM, the 
most popular resources, and gaps in the DREAM resources, in order to guide further design and content development. 
The primary data sources were site analytics, supplemented by an online survey and feedback from teachers who took 
part in a DREAM workshop. Results indicated that DREAM is steadily gaining users, and that the tool is effective and 
satisfying for music teachers as a way of keeping abreast about digital learning tools, especially for resources for ear 
training and sight reading. The paper concludes with suggestions for design enhancements and ways of broadening the 
user base. 


KEYWORDS 
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1. INTRODUCTION 

Digital applications for music education have been growing at an astonishing rate, and these resources are 
changing the ways people teach, learn, and make music (Burnard. 2007; Partti, 2012; Rainie & Wellman, 
2012; Waldron, 2013). While the use of digital tools in classroom music settings has a long history and is 
well described in the literature (Savage, 2012; Webster, 2012), the use of digital tools in studio music 
instruction is less prevalent (Upitis & Abrami, 2014). In both instances — classroom music teaching and 
private studio music instruction — obtaining information easily and swiftly about these new tools is important 
so that teachers can assess the appropriateness of such tools for their students’ needs in a timely fashion. 
However, teachers are perennially time-starved and are often unable to systematically examine and evaluate 
the resources that are available. Consequently, classroom teachers often rely on interactions with their 
colleagues to learn about new technologies (Savage, 2012). But independent music teachers often work in 
isolation (Feldman, 2010), making these informal discussions about resources more unlikely and certainly 
less than comprehensive. Thus, a tool that provides a centralized place where independent music teachers can 
keep abreast about new digital technologies for their field has the potential to assist music teachers 
considerably. It is this type of tool that was examined in the present study. 

The Digital Resource Exchange About Music (DREAM) is one of a suite of four digital tools for music 
learning. The first tool in the suite, iSCORE, supports students in their path to becoming self-regulated 
musicians. Two tools that are presently under development include an iOS mobile app for annotating music 
videos called Notemaker, and another web-based tool called Cadenza, which, like iSCORE, also supports 
student self-regulation. These tools were designed by a research and development team comprised of 
members from two universities and a national music conservatory (Upitis, Abrami, Brook, Troop, & Varela, 
2012). DREAM was designed by our team to encourage teachers to learn about digital resources related to 
learning to play a musical instrument following the Western musical tradition, both in classroom and studio 
settings. DREAM enables teachers to evaluate the resources, to read about other teachers’ views of the 
resources, and to add resources of their own to the digital learning objects repository. In the release version 
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(DREAM, v. 1.4), over 3,000 high-quality English and French resources were organized into six categories: 
(a) musical repertoire, (b) ear training and sight-reading, (c) practising, (d) history and theory, (e) creating 
and composition, and (f) professional resources. All of the entries in DREAM are searchable by title and key 
words, and users can also filter the resources by instrument, ability level, or platform (e.g., by type of tablet 
or smartphone). DREAM recommends resources to users based on their prior choices. We characterize 
DREAM as a Trip Advisor™ for music teachers — but instead of choosing a hotel based on such filters as 
free parking and breakfast, the teachers choose, for example, musical repertoire, filtering for instrument (e.g., 
piano) and difficulty level (e.g., advanced). In the four-month period immediately after launching the tool, 
the resource base grew to nearly 3,500 entries, the bulk of which are in the repertoire category. 

In this age of ubiquitous and accessible digital tools, it is essential that DREAM operates in a way that is 
seamless and efficient for intended users. Even though DREAM is essentially an educational tool, the 
expectation from users is that it will function as seamlessly and professionally as commercial products, such 
as the Trip Advisor™ described above. The process of usability testing involves learning from test 
participants that represent the target audience. These test participants help determine the degree to which the 
product meets its goals (Rubin & Chisnell, 2008; Yadrich, Fitzgerald, Werkowitch, & Smith, 2012). Rubin 
and Chisnell (2008) assert that a usable product must be “useful, efficient, effective, satisfying, learnable, and 
accessible” (p. 4). In a similar vein, Barnum (2011) states that tools should be easy to learn, easy to use, 
intuitive, and fun. Barnum notes that the International Organization for Standardization (9241-11) defines 
usability as, “the extent to which a product can be used by specified users to achieve specified goals with 
effectiveness, efficiency, and satisfaction in a specified context of use” (p. 11). Rubin and Chisnell embellish 
this definition of usability by noting that one of the most important aspects of making something usable is the 
“absence of frustration in using it . . . [so that] the user can do what he or she wants to do the way he or she 
expects to be able to do it, without hindrance, hesitation, or questions” (p. 4). The parallel idea in Barnum’ s 
(201 1 ) work is the notion that usability should be invisible, that is, the built-in usability of products suits the 
user so that the user doesn’t have to “bend to the will of the product” (p. 1). 

Before releasing DREAM to the public, an extensive usability testing protocol for DREAM was designed, 
and has been reported elsewhere (Upitis, Abrami, Brook, Pickup, & Johnson, 2015). During the first usability 
study, the DREAM design and contents were modified on the basis of the input of 12 core test participants 
and designers, a group of 24 classroom music teachers enrolled in a teacher education program, as well as 47 
studio music teachers representing eight of the thirteen provinces and territories in Canada. The results of the 
Phase 1 usability study gave support for the conclusion that DREAM could serve as a centralized place for 
music teachers to keep abreast about digital technologies, providing a tool that would be valued by music 
teachers in terms of efficiency and effectiveness. Further, the usability study provided evidence that DREAM 
would be enjoyable and efficient, so long as the team continued to vet resources, provide reviews, and 
otherwise ensured that only the highest quality resources would be included. 

DREAM (v. 1.4) was released on September 4, 2014 with a single email announcement to approximately 
30,000 users through the database of The Royal Conservatory (RCM), a national music conservatory in 
Canada. Most subscribers to the database are from Canada, although a small number live in the United States, 
where the RCM also conducts music examinations. Since that time, our data reveal that all new users have 
joined DREAM primarily by Internet searches and word of mouth. As recommended on the basis of the first 
usability study, a small team of music teachers has continued to vet new resources from outside users, 
provide reviews of the most popular resources, and add new resources as required to fill gaps, such as 
recordings for instruments other than the piano. Further, DREAM has been presented to various teacher 
organizations both for information and feedback, and several professional publications for independent studio 
teachers have featured short descriptions of DREAM. Overall, the first four months are best characterized as 
a “soft launch,” that is, a gentle introduction to the tool rather than a strategic marketing plan, the latter of 
which the developers will undertake after the results of the present study are incorporated in Version 2 of 
DREAM. Strategic methods for marketing digital tools have been in use for several decades (e.g., Downes & 
Mui, 2000; Edelman, 2010; Ryan, 2014). While most of these sources examine marketing strategies for 
businesses, it is clear that an educational digital tool, like DREAM, could be launched using similar 
strategies, some of which will be explored in the final section of the paper. 
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Thus, the purpose of the present study was to describe and analyze the first four months of use to guide 
further development of the tool and to identify strategies for expanding the user base. The research questions 
for this second phase determined: (a) patterns of use for DREAM (b) which pages and categories were most 
useful, (c) what gaps were identified by users and developers in terms of DREAM resources, and (d) what 
features of DREAM require further development to increase user satisfaction. 


2. METHOD 

2.1 Data Collection 

The primary sources of data were the site analytics tracked by Google, as well as by the host server. These 
site analytics included data on user numbers, page views, countries of origin, new and returning users, and 
information about the time spent on various parts of the site. There were two secondary sources of data: a 
focused survey and feedback from music teachers who took part in a workshop on DREAM. The focused 
survey was targeted to DREAM users through the project website as well as through Facebook. The survey 
contained one question regarding teachers’ use of DREAM to triangulate with the site analytics data 
regarding which of the six categories of resources were most often accessed. In addition, we asked a series of 
usability questions, including whether they shared DREAM with their students and colleagues, whether 
DREAM had changed their teaching, whether the resources were useful, and finally, whether there were any 
changes that they would make to DREAM in terms of content, function, and design. Feedback from the 
teachers taking part in the workshop focused on design aspects of DREAM, notably the browse and search 
functions. In total, 22 teachers provided comment through the survey or workshop. 

2.2 Data Analysis 

Open-ended questions from the targeted survey and workshop feedback were analysed according to the 
research questions described above. Descriptive statistics were compiled from the closed-ended survey 
questions regarding patterns of use. Site analytics were gathered and compared from the two sources to 
compile a portrait of user information as it pertained to the first four months post-release. 


3. RESULTS 

3.1 User Characteristics and Patterns of Use 

Over the four-month time period from September 4, 2014 to January 4, 2015, DREAM received over 40,000 
page views. Nearly a third of the users returned to the site after their initial exposure to the tool, coming back 
to the site both to access and to contribute content. Over the four months, nearly 5,000 new users accessed 
DREAM, mostly to view content. Users can view content without creating an account; however, to add 
content or to comment on resources in the digital repository, an account must be created. Of the new users, 
only 187 people (4%) created an account. This is not surprising, given that many users will simply use 
DREAM to locate resources and have resources recommended to them, and not be interested in adding new 
resources or reviews of their own. Earlier we characterized DREAM as a Trip Advisor™ for music teachers; 
many people who use Trip Advisor™ to make travel plans never provide a review upon their return. 

As can be seen in Table 1, users of DREAM typically spend just over three minutes on the website, 
browsing six pages on average. This means that they go well beyond the home page, and likely search or 
browse until they find the resource they are seeking. There is also a difference between new and returning 
users. New users access approximately three pages, while returning users engage with six or more pages each 
time. Thus, it appears that users who find DREAM helpful return each time for longer and longer sessions. In 
addition, this result does not include the time that they may spend learning more about the resources that they 
locate through DREAM. In the workshop with teachers, we observed that teachers would browse through 
two or three resources, and. finding one that piqued their interest, they would then leave DREAM and spend 
time examining the resource they had identified on the webpage dedicated to that particular resource. 
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Table 1. Overview of site analytics 


Measure 

Unique sessions 

6,778 

Users 

4,788 

Page views 

40,091 

Pages visited per session 

5.91 

Average session duration 

03:28 minutes 

New users 

70.1% 

Returning users 

29.9% 


The site is predominantly accessed via Windows operating systems (45%) followed by iOS mobile 
devices such as iPhones and iPads (25%), Macintosh computers (19%), and Android mobile devices (8%), 
with the remainder being accessed through BlackBerry, Linux, Nokia and Windows Phone. The most popular 
browser is Safari (34%), followed closely by Chrome (30%). Firefox and Internet Explorer combined make 
up another 25%. Nearly two thirds of the visitors are from Canada (64.78%), with a notable number from 
other English-speaking countries, such as the United States (13.93%), and the United Kingdom (2.15%). 
Some users are from other countries in Europe, as well as Hong Kong. The users are predominantly English- 
speaking (83%); fewer than 2% are currently accessing DREAM in French. 

As noted previously, the resources in DREAM are divided into six categories. In descending order of 
popularity, the resources accessed were (a) musical repertoire, (b) ear training and sight-reading, (c) 
practising, and (d) theory and history. The remaining two categories, namely, creating/composing and 
professional resources, were rarely accessed, indicating that these resources were less important to teachers. 

These patterns of use correspond to the results of the survey and feedback from the teacher workshop. 
Every teacher we observed in the workshop, and every survey respondent reported that they used DREAM to 
find ear training and sight-reading programs and apps. While repertoire had more page views, the number of 
page views of ear and sight resources is proportionately much higher. Near the end of the four-month period 
that this study took place, there were 2,964 repertoire resources in DREAM, compared to only 44 ear and 
sight resources. This means that 85% of the resources were repertoire pages, and a mere 1.25% of the 
resources related to ear training and sight-reading — yet they were accessed nearly as often as repertoire. Put 
another way, in terms of the total page views, repertoire accounted for 5.24% of the page views, and the ear 
and sight resources represented 4.40% of the page views. The importance of the ear and sight resources to 
teacher pedagogy was also reflected in the open-ended comments on the survey. One teacher noted, “The ear 
training and sight-reading apps I’ve located have been particularly handy. . . I also extolled its virtues in an 
article about the sight-reading apps!” 

3.2 Further Survey and Workshop Results 

The site analytics do not give a direct indication of whether the resources recommended by DREAM are 
useful to users, although certainly the high rate of returning users is an indirect indication of user satisfaction. 
Consequently, to learn more about usability, survey respondents were asked directly about whether DREAM 
resources were useful, and every respondent answered “often” or “sometimes”, with over 60% selecting 
“often.” The categories of rarely and never were not used. Another indication of user satisfaction from the 
survey results was that every respondent indicated that they have told colleagues and/or students about 
DREAM. As to changes in pedagogy, most respondents indicated that it was too soon to tell, although those 
respondents who indicated that DREAM has improved their teaching commented on the usefulness of ear 
training and sight-reading apps, repertoire, and instructional videos about learning. One user commented on 
the importance of peer reviews of the resources. 

At this stage in the development of DREAM, it is important to ascertain whether there are design changes 
that should be incorporated in Version 2. If users have concerns about DREAM, they tend to center on the 
search function. Some users are confused about the difference between browse and search; others have 
requested a more sophisticated search function that would enable them to make greater use of DREAM. A 
few users have suggested that repertoire should support more platforms (DREAM presently supports 
YouTube, but not, for example, Vimeo). 
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Other directions for improving the design of DREAM and increasing the resources include working 
closely with music conservatories worldwide to identify additional resources of both global and local interest. 
Asian, African, and Middle Eastern music are not well represented in DREAM and this is also a way to 
enhance the repository. In addition, seldom used and poorly rated resources might eventually be held aside. 

Finally, the site analytics indicate that most users access DREAM directly, using the URL to visit the 
website. The remaining users often find the site using search engines. A small minority of users come from 
site referrals and social media. These results will guide our strategies for increasing the DREAM user base. 


4. CONCLUSION 

For a specialized tool for music educators that was launched in a low-key way, the first four months of use of 
the DREAM tool was impressive. The number of page views, especially for resources in the ear training and 
sight-reading category, was considerable. Many teachers also accessed the apps and other instructional 
resources designed to support student practising. This is not surprising, given that teachers often identify that 
one of the perennial challenges of instrumental music instruction is to guide student practice and to provide 
scaffolding for students to become proficient sight-readers (e.g., Hallam et al., 2012). The results from the 
present study, which point so strongly to the importance of these types of sight-reading and practising 
resources, will help us focus on new content in these categories. Given the results, other design changes 
would also be prudent, especially with respect to the search and browse functions. We expect that the 
improvements for the search function in particular will have a positive impact, given that the search pages are 
accessed much more frequently than the browse function and that teachers have commented on specific ways 
that the search function could be improved. 

The results also indicate there is potential to grow the user base. There are at least three reasons for this 
conclusion. First, there has been a steady increase in users each month, even without strategic marketing to 
reach out to more teachers. Second, this organic growth of DREAM includes a substantial group of users 
(35%) who reside outside of Canada, and therefore are not among those who would have received 
notification the release of DREAM. Finally, there are clear indications that the tool is efficient, satisfying, 
and useful to users. 

How, then, to grow the user base? While evidence has shown that word of mouth and direct linking have 
been the most effective ways of expanding the DREAM user base, there is merit in considering other sources 
of traffic. Search traffic from Google and other websites has accounted for a majority of the site sessions to 
date. By promoting our search results with the use of Google AdWords, and SEO optimization through smart 
site linking in reputable and related sources, our target audiences would encounter DREAM more often. 
There are also web-based music forums that may be worth investigating for promotional value. Having active 
exposure on these forums might increase referrals through direct links as well as word of mouth. At present, 
Facebook is our sole provider of social media referrals. Facebook contains targeted ads that can be focused 
on specific demographics. By setting up these targeted profiles, we can ensure that DREAM exposure 
increases to music teachers, students, and other interested parties. 

Edelman (2010) claims that successful digital marketing has four components, two of which have direct 
application to DREAM. He argues that marketers must first “engage the consumer throughout an 
increasingly digital purchase journey” (Edelman, 2010, p. 2). This strategy suggests that continuing to grow 
the user base, without charge, in the ways discussed above. Over time, adding such features as a donate 
button and creating a premier version might serve to both increase the user base and generate revenue to 
maintain and enhance DREAM. Edelman (2010) also stresses that successful marketers continually gather 
and use digital data to guide their actions, and this we will also continue to do with the aim of refining 
DREAM and making it available to the widest possible audience. 
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ABSTRACT 

The purpose of this paper was to examine the unique areas of concern when establishing an eLearning program in the 
field of global business. A survey of eLearning and a global management subject matter appears. This paper identifies 
potential challenges in program design and raises practical concerns for future research. 
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1. INTRODUCTION 

Workforces are becoming more global (Remtulla, 2007a) and educational instructions are moving more 
towards non-traditional learning ("Online enrollment", 2010). Since the use of a delivery method can 
significantly shape the success of a program, the success of graduates, and the success of an institution's 
reputation, it is important that the choice delivery method be examined thoroughly. As different programs 
within an institution have distinct subject matter that needs to be uniquely presented and tailored it is 
important that the design and delivery of each topic of study is looked at individually. 

The purpose of this work was to add to the breadth of research available by examining questions that 
should be answered in a global business master's program delivered online. The project also calls to attention 
a large gap that existed in available research material. Information on pedagogy, eLearning, the global realm, 
and graduate business programs as isolated areas is plentiful. However, finding material that ties these areas 
together or focuses on the course subject matter is not an easy find. 

A global business program may also be known as names such as an international business program, a 
global management program, or similar title. The program may appear as a solo offering or as a 
concentration within another field (such as an MBA program with a global business concentration). This 
paper will not distinguish; it will focus on courses that are solely delivered via an eLearning format. 

If there are areas of global business management that cannot be taught via eLearning educational 
leadership will need to know this. If there are areas that are difficult to teach, educational leadership will need 
to know what techniques are effective. Limitations need to be examined honestly as there will be 
ramifications on the success of the program. An institution should also be proactive in this search for 
information as finding a problem out reactively could mean the problem is widespread and debilitating to the 
institution. 


2. EXAMININGELEARNING 

Rosenberg (2001 ) defines eLearning as, "the use of internet technologies to deliver a broad array of solutions 
that enhance knowledge and performance". Rosenberg further states that eLearning is networked, delivered 
via a computer and standard internet technology and that it is focused on the broadest view of learning. 
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eLearning is a popular, but smaller segment of the field of distance learning. Distance learning can involve 
areas such as media tapes or delivery via standard mail. A key to eLearning is the replacement of 
face-to-face interaction with a hardware interface (Remtulla, 2007b). 

A survey by Babson Survey Research Group study revealed, in 2013 online students numbered 7.1 
million, a 6.1% increase from the previous year, and the lowest recorded increase in a decade of steady 
increases. 33.5% of higher education students reported taking at least one online course (Allen & Seaman, 
2014). 

eLearning can provide students with an opportunity to attend courses without altering their existing 
schedules, or moving house. Students can pick their most convenient times to study (within some restraints) 
and have a better chance of maintaining work-life-school balance (Li, 2008). Individuals who do not feel as 
though they are being developed in the workplace may turn to eLearning at a university for a form of 
self-development in hopes that it may set them up for a future position. 

Desai and Pitre (2009) list several advantages to offering a course online. These include the time 
flexibility for students, the chance for a school to increase enrollment, the chance for students to have diverse 
group interactions with students from other cities and countries, possible exposure to unique culture and 
language, a wider perspective of problem solving, and project variety. 

Chau (2010) acknowledges that cost, accessibility and flexibility may be motivations for institutions 
moving to online learning. However, Chau also proposed that educational institutions are motivated by 
providing a commodity in current demand in the consumer marketplace. 


3. CHALLENGES OF ELEARNING FACILITATION 

eLearning is not without challenges. For facilitators there can be isolation, time demands, highly rigid 
courses, and a lack of traditional student/teacher hierarchy. 

One unique challenge of eLearning is a staff that may be external to the university campus. Instructors 
may feel a sense of isolation from the university or program itself (Magdalena & Mellar. 2009). Instructors 
may have little vested interest in student success. The understanding of university support areas may be 
limited. Instructors working on a contract or ad-hoc basis may also be responsible for reporting or fixing the 
problems found within the course (Magdalena & Mellar, 2009). 

Students may also experience isolation. Students did not miss the opportunities for social interaction 
when taking courses online (Mallory, 2007; Smith & Duus, 2001). However, in a traditional setting students 
may feel a closer bond to the instructor (Lucas, 1996). Smith and Duus (2001) stress the need for small class 
sizes due to the intense dialogue interaction that is needed between student and instructor. Smith and Duus 
(2001) further note that although offering a class online may result in cost savings due to increased 
opportunity, the small class sizes may make that savings null. 

An additional challenge to instructors may be the rigidity of the course. One study found that "the 
demands for a structured and focused approach to virtual interactivity may be too much of a challenge for 
some" (Smith & Duus, 2001). Although the course itself may be more rigid than a traditional course, 
instructors may find the student/teacher hierarchy within the course much less so. A course with open 
discussion forums may also blur the line between the instructor and peers in interaction. Instructors need to 
learn how to be leaders in the virtual world and build a sense of community and collaboration. Engaged 
faculty can help students feel connected (Cook, 2010). However, instructors who move from a traditional 
course to teaching online may find the time demands much more difficult, as students may expect quick 
answers to questions at any hour (Cook, 2010; Smith & Duus, 2001). Du-Charme-Hansen and Dupin-Bryant 
(2005) emphasize the need to build a distance education plan that involves actively building community, 
facilitating communication, and humanizing the experience. Many of the suggestions have to do with course 
threads, synchronous chat and engaging in more informal conversation about "how's it is going?". Although 
an active classroom discussion board is emphasized in the literature the desire for synchronous sessions 
remains under debate (Cunningham, 2014; Nicholas, 2014). 

One study suggested universities focus on course design, instructor selection and instructor involvement, 
as the delivery method of the course did not have a statistically significant impact on learning (Buhagiar & 
Potter, 2010). Another case used the example used the social interactive website. Second Life, as an 
instructional tool to improve faculty engagement and to develop a sense of community among faculty and 


158 



International Conference e-Leaming 2015 


students (Sutton, White, Mbizo & Stewart, 2010). Verene (2013) argues that complete learner-instructor and 
learner-learner engagement cannot take place online due to the lack of genuine, real time dialogue and 
communication nuances. 


4. CONTENT DESIGN FACTORS 

There are several factors that should impact how a course is crafted and presented. Among these are course 
content, pedagogy, the subject matter covered, the learners, and the media to be used. 

Edmundson (2009) found that course outcomes affect the course content, pedagogical approaches, and the 
type of media used. Some courses such as computer courses tend to be culturally neutral. Other areas such 
as leadership courses were "deeply imbedded with cultural values, ideology, and worldviews" (Edmundson, 
2009). As the content was more complex, the pedagogical approaches and media needs become increasingly 
complex as well. Making the content accessible to a culturally diverse audience may include examination by 
observation, focus groups, country experts, interculturalists, web interface designers and a close look at the 
targeted learners (Edmundson, 2009). eLearning courses should be tailored based on the content and the 
learners unique identity and needs. 

Universities may or may not examine the subject matter of a course when deciding whether a class should 
be offered online. There may be other considerations, such as space concerns, instructor pool, or the need to 
be geographically diverse to get a wide enough student base. The decision to offer a course online may be 
solely administrative, yet instructional input will need to be offered. The instructional designs and technology 
chosen should align with the course goals and pedagogical goals (Cook, 2000). In order for this alignment to 
take place the specific subject matter area needs to be examined. 


5. GLOBAL MANAGEMENT DESIGN 

In global management subject matter design, the natural place to look for program outcomes is competencies 
required by organizations that employ global management professionals. Much of the research focused on 
competencies needed function as an expatriate employee (Biicker & Poutsma, 2010). However, some 
competencies discussed were the ability to use social capital and diverse networks in a way that completes 
work effectively (Makela & Suutari, 2009). 

Criticisms of MBA graduates include that they are "ill prepared to deal with complex, multi-layered 
issues faced by managers in global markets" (Belasen & Rufer, 2007) and that the theoretical courses that 
they take result in too much emphasis on the quantitative and not enough emphasis on people skills 
(Sulaiman & Mohezar, 2009). Tay (2001) suggests that courses such as public speaking, conflict resolution, 
and team work techniques be added to be business school programs in an effort to enhance soft skills of 
students. Bueno and Tubbs (2004) suggest that the most important global leadership competencies are: 
communication skills, motivation to learn, flexibility, open-mindedness, respect for others, and sensitivity. 
These are soft skill areas and may be areas difficult to teach or develop in students. 

Over the last thirty years business schools have been forced to react to an AACSB mandate to 
internationalize their curriculum. Some schools have reacted by merely adding an international business 
survey course. Others have integrated an international component into existing subject areas such as 
marketing, accounting or management. Some programs insist on going beyond mandate minimums in 
program design (Delaunay & Blodgett, 2005). 

Some traditional universities are able to take unique advantage of opportunities around them to develop 
competencies. Others make a conscious effort to create international core community within the university. 
Suffolk University developed a Global MBA (GMBA) program with the idea that it would tap into 
opportunities in the vibrant international business community around Boston. The program makes a 
conscious effort to ensure that a global perspective is always in place within the classroom. One third of the 
faculty members of the Suffolk University's business school were born in and educated outside of the United 
States; half of the faculty base has extensive experience in work or education abroad. The University 
maintains several campuses abroad. Fifteen percent of the students of the school of business come from 
abroad (Delaunay & Blodgett, 2005). MBA students at Oxford spend six months of their program with 
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organizations located in other countries (Tay, 2001). Some universities offer a regional world focus when it 
comes to courses or offer some specializations or seminars abroad (such as finance in Japan) (Delaunay & 
Blodgett, 2005). 

With more business practitioners becoming consultants than in the past, offering a course preparing 
students for consulting activities may have advantages. Students could learn about consultant 
communication, research, assessment and relevant business communication (Dallimore & Souza, 2002). 

Foreign language requirements may be needed by those who would like to do business in the global 
arena. Freeman (2001 ) notes languages are needed for a broader range of job functions than in the past. 
Freeman also notes that more small firms are taking advantage of international business opportunities than 
ever. Staff that can communicate effectively is needed. 


6. IMPLICATIONS FOR FUTURE RESEARCH 

The lack of material available in the design of business curriculum and global business course development 
makes it possible to suggest areas for future research. Research that makes connections between the topics of 
adult learning, online delivery, a global business subject matter, and graduate level students is proposed. 
Programs currently exist that can be examined. Effectiveness can be looked at from a student perspective, 
administrative perspective, and business perspective. 


7. CONCLUSION 

Global management and eLearning were both found to be areas of growth. This study was a rare attempt to 
look at the global management within a graduate eLearning context. The need for further research was 
evident and some areas of future, specific study proposed. The areas of future examination include 
administration, instructors and students. Considerations of program validity, reliability and overall evaluation 
should be considered at the time of program creation. 
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ABSTRACT 

In this study we presented new EES Model-2 extended from EES model for more productive implementation in 
e-learning process design and modelling in higher education. The most updates were related to uppermost Instructional 
layer. We updated learning processes object of the layer for adaptation of educational process for young and old people, 
taking into account interests and abilities of students of the different age groups. Important added objects of the 
uppermost layer are cultural diversity and language. We updated communication object of the Instructional layer and 
added human and social factor. The methods of study of Instructional layer were updated with selective object “religion”. 
We added to E-paradigm layer an object “combination” that explains by combination of synchronous and asynchronous 
objects. 

Our study has high significance for increase of quality of e-leaming in higher education in specific cases. We strongly 
recommend application of this updated EES Model-2 to support high educational standards of higher education and 
provide rights of students with different needs and abilities. 
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1. INTRODUCTION 

Because enabling technologies present many opportunities as well as challenges in the realizing of electronic 
learning (e-learning), it is imperative that educators and institutions planning to embark on the development 
of e-learning systems, have a clear and accurate understanding of the capabilities, limitations and influences 
of these technologies (Cloete, 2000). Creative approaches and competent strategies to manage these 
limitations at the instructional design, the user levels as well as integration to other systems, need to be 
established and understood in order to ensure a degree of quality comparable to that of traditional learning. 
Without the integration of well-established methods and techniques, many of the e-learning efforts may be 
futile, leaving frustrated facilitators and badly educated students in their make (Cloete, 1999, 2001 ). 

The creation of an e-learning system needs to have a model. The first generation of e-learning system was 
to manage and measure the learning process, display some kind of learning objects but they didn’t deal with 
reusability and organization. These were the Learning Management System. The second generation 
electronic learning systems, based on Ismail (2002), has to be able to manage searchable, reusable and 
platform-independent learning objects. Cloete (2001) has improved the system and developed a layered 
model for second-generation e-learning systems: Electronic Education System (EES) Model. The aim of the 
model is to assist the designers of different e-learning settings to plan and implement a specific learning 
situation, with the focus on the individual requirements and milieu of the learning group (Cloete, 2001). The 
multilevel EES model contains four layers (Fig. 1, Cloete, 2001 ). 

IEES model 

[has! 

four layers 

iaL 

| Instructional layer | 

| Educational middleware layer | 

| E-paradigm layer | 

| Physical layer 1 

Figure 1. Four-tier model for Electronic Education System (modified after Cloete, 2001) 
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These layers are strictly separated in their functions and each layer uses the services of the lower level 
layers (Dulai et al, 2013). The strategic development of e-learning can be carried out either on top-down or 
bottom-up manner, or as combination of both (Gullu et al, 2014). These approaches were implemented by 
many universities. Their target is application of the potential of e-learning to enhance teaching and learning. 
In addition, staff training is seen as essential to successful e-learning but flexible support structures and 
mechanisms are seen as even more important (MacKeogh and Fox, 2009; Drlik and Skalka, 2011). 

The need to update existing EES model raised from the modern issues influenced on educational process 
in our society. We explored that diversities of age, religion, language, culture are making significant 
influence on educational process of current generation of students. According to number of studies based on 
personal interviews and detailed research we found that current students can be older, more religious and 
with strongly marked commitments to language, culture and nationality (e.g. Stolzenberg et al, 1995; Myers, 
1996; Sherkat, 1998; Sherkat, 2007; Terry and Irving, 2010; Cavazos, 2015). 

In this study we extended the EES model, explaining in details each layer, and presented new EES 
Model-2 taking into account described issues. 


2. EES MODEL-2 STRUCTURE 

The basis for extended EES Model-2 is EES model developed by Cloete (2001). This is a model where can 
be implemented a top-down and a bottom-up algorithm approaches to design of a strategic model for a 
particular e-learning situation. Cloete (2001) described in detail implementation of these approaches and 
basic design of the EES model. In this study we updated some layers adding new elements into the EES 
Model-2 and described them in detail. 

2.1 Instructional Layer (Uppermost) 

The purpose of the instructional layer is to serve as a window between the learning process and the 
underlying strategies necessary to establish the learning environment. The instructional layer is composed of 
various objects, each containing one or more methods (Cloete, 2001). In our EES Model-2 the Instructional 
layer consists of intermediate elements. Learning process and Learning environment strategies, and main 
objects (Fig. 2). The Learning process can be adopted for young and old people, who have different needs 
and ways for study. The main objects are containing different methods of study (by watching, reading, 
discovering, observing, listening, doing and cooperative learning). We added religion into the methods of 
instructional layer as an important object, which can strongly motivate students for seeking of knowledge in 
countries where religion has big importance (i.e. Turkey, Arabic countries, Malaysia and Indonesia). This 
element is mentioned as selective due to its specific implementation. 

The main object element contains of communication objects and objects of content. The communication 
objects describe differences of students by social and human factors. Human factor means that every person 
is individual and specific approaches can be implemented for different persons. We found that social factor is 
important element in Turkey. Turkish people are very sensitive for social status of different persons and 
respective environment must be applied in such cases. 

Objects of content describes cultural and language differences. We analysed cultural and linguistic 
situations in Estonia and Turkey and found that these elements have high importance for e-learning. 
Implementation of cultural element and language preferences of different groups of students into the 
e-learning environment will increase interest for education and motivate students of different cultural and 
linguistic societies for study. 

2.2 Educational Layer (Middleware) 

The educational middleware layer provides services for a reliable and effective learning environment (Cloete, 
2001). It contains (1) user authentication, (2) assignment, (3) course enrollments and (4) testing services. 
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2.3 E-paradigm Layer 

The objective of the e-paradigm layer is to provide an electronic learning paradigm composed of 
technological strategies possible in electronic learning. The objects found on this layer form the basis of the 
specific learning situation. They often prescribe which objects from upper layers may be suitable for 
selection (Cloete, 2001). The E-paradigm layer 2 represented by “Possible technological strategies” 
(synchronous, asynchronous and combination). The synchronous and asynchronous objects are commonly 
identified on the e-paradigm layer. In synchronous learning environments geographically dispersed, students 
and lecturers share a virtual classroom within the same physical time frame. Examples include remote lecture 
rooms with video conferencing, or students attending real-time lectures from home. The asynchronous object 
is characterised by its being independent of location, time, and learning speed of the learner. A typical 
example is that of the learner who prefers to study at his/her own pace and time. The number of methods for 
objects on this layer is limited, and is realised on other levels. For example, selection of the asynchronous 
object will have a direct influence on the methods of the course distribution object found on the educational 
middleware layer. Methods may be through web downloads or precompiled CDs while in the synchronous 
environment, e-books and on-line material may be more relevant (Cloete, 2001). In our EES Model-2 we 
added to this layer an object “combination” that explains by combination of synchronous and asynchronous 
objects. An example is that learner who has unstable time schedule on his job has opportunity to choose and 
combine between two main ways of study: synchronous (to study in the same time frame with lectures 
attending real-time lectures) and asynchronous (to be independent of location, time and speed of the learning 
process). 

2.4 Physical Layer (Bottom) 

The physical layer provides for the transparent transmission of messages (which may be course 
communication, course material or course directives) between students and lecturers tied together in an 
e-learning scenario. The physical layer includes the specification of hardware and software technology 
objects necessary to accomplish e-learning. The number of methods included in these objects is usually 
limited to one but may sometimes extend to two. For example, an object on this layer may be an Internet 
connection. The methods of the Internet connection object describe the prerequisite hardware and software 
strategies necessary to accomplish an Internet connection. The Physical layer of the EES model was extended 
in the EES Model-2 to: “Government purchased devices” (e.g. laptops, tablets), “BYOD” (Bring Your Own 
Device, multiplatform, single platform), “computer laboratories”. 

2.5 Evaluation Plane 

An evaluation plane stretches across the top two layers. This plane performs evaluation functions related to 
these two layers as a whole. The purpose of the evaluation layer is to determine whether or not the methods 
selected from the instructional layer and from the educational middleware layer are accomplishing the 
established goals and objectives. The evaluation plane is divided into a summative evaluation sub-plane and a 
formative evaluation sub-plane. Formative evaluation is typically conducted during the lifetime of a process, 
whereas summative evaluation is conducted at the end, or after the lifetime of a process (Wills 1995; Cloete 
2001). In an e-learning system, one may for example choose to do both types of evaluation and must then 
include objects from both sub-planes, or one can include only one type of evaluation, analysing one's 
learning situation through various methods (from selected objects) as found in that particular sub-plane. More 
detailed description of evaluation plane can be found in Cloete (2001 ). 
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3. DISCUSSION 

The Cloet’s (2001) EES model was updated and extended in this study (EES Model-2). We found that 
several factors, such as student’s age, differences of students by social-cultural and human factors, language 
differences and religion were not included into EES model. According to many studies in the field (e.g. 
Sherkat, 2007; Terry and Irving, 2010; Cavazos, 2015), nowadays these issues are very actuals in modern 
society and have significant influence to education and higher education in particular. Thus we added them 
into the EES Model-2. This updates are important for further studies related to e-learning process modelling. 

The most updates were related to uppermost Instructional layer. First we updated Learning processes 
object of the layer taking into account interests of students of the different age groups. For example, older 
people are more conservative in implementation of new technologies into to their life and need more time for 
adaptation to the new environment. 

Other important objects of the uppermost layer are cultural diversity and language differences. Cultural 
preferences must be taking into account during e-learning process design. In Turkey customs are very 
important part of life of young people and can be used to design more productive educational process. For 
example, respect of old people and teachers is still common in this country. 

It was found that language differences factor is very important and sensitive for young and old students in 
both countries, Turkey and Estonia, which are very different. Thus, we highly recommend apply this object 
in e-learning design. 

Next updates were related to communication object of the Instructional layer. We added human and social 
factors to be implemented in e-learning modelling in the part of communication specifications. Human factor 
contains individual specific needs of every student. Taking into account this factor the efficiency of the 
e-learning can be increased. For example some students can be more familiar with some objects of study, 
other need more time for explanation. Or some students are very flexible for change of software environment 
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and other needs more time to adopt. The social factor is important element in Turkey. Turkish people are 
very sensitive for social status of different persons. It is not a rule in Turkey, but, for example, we found it 
very often that young people grouping into clusters by social status and don’t allow access this groups for the 
people of lower social class, as people came from villages or from families with low income. In such 
situations, to support more effective education, it is recommended to provide different virtual classes for such 
students. 

We added religion into the methods of study of instructional layer. This element is mentioned as selective 
due to its specific implementation. Religion can strongly motivate students for seeking of knowledge in 
countries where religion has big importance. Islamic religion prescribes and motivates all age people to learn 
and discover the world in all the ways. This factor has absolutely importance in such Islamic countries as, 
i.e., Turkey, all Arabic countries, Malaysia, Indonesia, some African countries, etc. From the other side some 
authors (e.g. Sherkat, 2007) found that there is a fundamentalist Christianity problem in our society, in 
American society in particular. According to this study, young sectarian and fundamentalist Christians often 
have difficulty dealing with environments. E-learning will ease educational process in this particular case. 
Anyway, e-learning system model have to take into account this factor for societies, where this problem 
exist. This issue is case specific and preliminary explorations must be provided in each particular study. 

Our study has high significance for increase of quality of e-learning in higher education in specific cases. 
We strongly recommend application of this updated EES Model-2 to support high educational standards of 
higher education and provide rights of students with different needs and abilities. The new developed EES 
Model-2 will be used for our future work to enhance quality of e-learning in higher education in particular 
countries (like Turkey), as well as in the field of study in general. 
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ABSTRACT 

Although used as a common pedagogical tool in K-12 education, online graphic narrative (“comics”) software has not 
generally been incorporated into advanced professional or technical education. This contribution reports preliminary data 
from a study on the use of cloud-based graphics software Pixton.com to teach basic medical ethics concepts and the 
human dimension of medical ethics dilemmas in a premedical program at an American medical college in the State of 
Qatar. Qualitative and quantitative end-of-semester data (open-ended and 7-point Likert scale questions) were collected 
from randomized control and study groups from 22 total students in a 4 -week medical ethics module, half of whom wrote 
an argumentative essay on a contemporary medical ethics dilemma and the other half who created a 10-15 panel graphic 
narrative consisting of a real-life doctor-patient-family encounter involving a contemporary ethical issue. 


KEYWORDS 

E-leaming; Cloud computing - graphic narrative software; narrative medical ethics; medical education; Qatar. 


1. INTRODUCTION 

The study aimed to provide preliminary data for larger sample size randomized controlled trials investigating 
student learning about medical ethics questions by comparing two pedagogical modalities: a argumentative / 
persuasive research paper and a graphic narrative using a cloud-based online graphics software program 
called Pixton.com. The research attempts to answer the general question about the effectiveness of using 
graphic narratives (stories) in order to help students understand the human dimension of biomedical 
conundrums that they will eventually face in their professional practice. A total of 22 students participated in 
a 4-week medical ethics module and then qualitative and quantitative end-of-semester data was collected 
from the students via questionnaires. 


2. BACKGROUND 

The narrative medicine movement began in the 1990s and seminal overview treatments of the topic by Rita 
Charon and Brian Hurwitz appeared a decade later, such as Charon’s Narrative Medicine: Honoring the 
Stories of Illness (2006), and Hurwitz’s Narrative Research in Health and Illness (2004). Narrative medicine 
is a clinical and research perspective in the healthcare field that investigates how stories (narratives) structure 
medical experiences, ranging from formalized case histories, ethics dilemmas, the patient experience of 
disease, and events in the medical workplace. Narrative medicine is an important and growing branch of the 
medical humanities (medical sociology, medical history, literature and medicine, etc.) which similarly seeks 
to explore humanistic elements in medical care both as an end in and of itself, and as a means to improve 
medical outcomes and patient satisfaction (Weber, 2010). In an editorial in the Australian Journal of 
Medicine, Jill Gordon aptly pointed out: “To appreciate whether the humanities are indeed foreign to 
medicine, try to imagine a health care facility in which no ethical issues are explored, no lessons have been 
learnt from the past, no cultural awareness is displayed, no written words (other than technical 
communications) appear, and no books, films, television programs, plays or concerts are discussed by 
patients or staff. Imagine that there are no artworks, no music and no other aesthetically pleasing elements. 
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Although some of our hospitals are admittedly run down, the products of the arts and humanities are 
nevertheless all around us” (2008, p. 420). 

The author’s experience in working in the Persian (Arabian) Gulf at an American medical college has 
reinforced the realization that medical practices, although grounded in the shared biomedical model of 
disease dominant in wealthy industrialized countries, can vary considerably from culture to culture, 
particularly with respect to medical ethics, doctor-patient relations and the patient experience of disease 
(Weber, 2009). Therefore the stories that both patients and medical practitioners tell about disease and their 
lived experiences are impacted by a range of socio-culture, religious, and economic factors. From a 
pedagogical perspective, the medical humanities has three key characteristics in medical education, according 
to Shapiro et al. (2009): 

1. They use methods, concepts, and content from one or more of the humanities disciplines to 
investigate illness, pain, disability, suffering, healing, therapeutic relationships, and other 
aspects of medicine and health care practice. 

2. They employ these methods, concepts, and content in teaching health professions students 
how to better understand and critically reflect on their professions with the intention of 
becoming more self-aware and humane practitioners. 

3. Their activities are interdisciplinary in theory and practice and necessarily nurture 
collaboration among scholars, healers, and patients. 

Graphical imagery is immediate, visceral and can communicate different kinds of information (and is 
especially effective at expressing human emotion), and has particular practical use for illiterate populations in 
conveying meaning. Therefore graphical language is pervasive in modern society, from advertising, the 
Internet, and all forms of print media as well as embedded within the design of objects. The nexus between 
images and story-telling may stretch back to the Paleolithic cave paintings of Lascaux in Montigac, France 
which possibly depict a narratological and even cinematic sense of reality (Noxon, 1964). William Flogarth 
( 1697-1764) is generally credited, however, with developing modern sequential art, also known as a comic or 
cartoon. His often reproduced engravings A Rake ’s Progress , A Harlot ’s Progress and Marriage a-la-mode 
take on a genuine moralistic tone satirizing libertinism, prostitution, and marriage for financial gain; thus he 
established the genre of sequential art as serious social commentary and as a didactic form of expression. He 
also was interested in medical topics (Haslam, 1996). Various forms of sequential art developed into the 
modern period, from the purely humorous comics often found in daily newspapers, to related forms which 
combine parody, satire, and social commentary (such as Doonesbury, Pogo, and the political cartoon) to 
book-length illustrated stories such as Art Spiegelman’s Maus (depicting his father’s survival of Auschwitz 
and its effect on his son) and Joe Sacco’s Palestine, an exploration of the social plight of Palestinians. 

Graphical techniques are also beginning to impact diagnostic and therapeutic techniques in medicine. For 
example. The Pictorial Representation of Self and Illness Measure (PRISM) employs simple graphical shapes 
such as circles in order to measure levels of pain and suffering. It has been validated and applied to a variety 
of patient populations, such as fatigued cancer patients (Gielissen et al., 2013), vertigo sufferers ( Weidt et al., 
2014), and victims of orofacial pain (Streffer et al., 2009). 

Since the range of topics which occupy modern graphic artists encompasses all of human experience, it is 
not surprising that a substantial body of graphic narrative specifically related to illness, disease and healing 
exists. The most frequent topics that have been dealt with recently are cancer and mental illness, undoubtedly 
due to the complexity of these diseases and the vastly different responses of families and sufferers when 
facing these conditions. Representative book-length examples of this genre include Marisa Marchetto’s 
Cancer Vixen (2006), an autobiography ( March etto is a professional cartoonist) that describes her personal 
struggle with cancer, and Brian Fies’s Mom 's Cancer (2006) which focuses on the effects of the disease on 
family members. Other notable medical graphic narratives include: Lesley Fairfield’s Tyranny about anorexia 
nervosa; Tucky Fussell’s Mammoir: A Pictorial Odyssey of the Adventures of a Fourth Grade Teacher with 
Breast Cancer, David B.’s Epileptic, and Darryl Cunningham’s Psychiatric Tales, another autobiographical 
work detailing his employment in a psychiatric ward. 

Pixton online comics can easily create the kind of graphic narratives that have appeared from professional 
artists; the comics are built in frames which consist of user controlled pre-defmed background templates. 
Background images such as .jpgs can also be imported and over-laid in various ways. In addition to objects, 
character forms from templates can be introduced into the panels and posed in various configurations using 
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pivot points that can be grabbed by the cursor, normally along the body joints. Most importantly, many 
aspects of the characters’ faces can be configured by selecting predefined elements and then dragging these 
elements to resize them. Thus, a wide variety of emotional responses can be custom drawn with Pixton. Two 
similar simple graphics tools include a desktop application by Plasq called Comic Life 
(http://plasq.com/downloads/comic-life-desktop/) and the cloud-based Make Beliefs Comix 
(www.makebeliefscomix.com). As Shamburg points out, “these programs simplify the more technical 
aspects of comic creation and let a user focus on the creative aspects of production. These programs allow 
you to save, edit, revise, and share your work” (2014, p. 2). 

A growing number of research studies are reporting successful educational outcomes using Pixton in the 
educational setting. In a technology graduate-level seminar, Kovalik et al. demonstrated increased 
engagement when students used Pixton as an advanced organizer before engaging in online discussion, i.e. in 
a flipped-classroom modality (2011). In addition, Meyers (2014) introduced creative practice via Pixton into 
an undergraduate course in communications theory. 


3. STUDY OBJECTIVE 

The purpose of this preliminary study, to be followed by a larger randomized controlled trial, was to compare 
differences in learning outcomes related to basic medical ethics concepts in a premedical department 
following an American medical curriculum (Weill Cornell Medical College in Qatar, a satellite campus of 
Weill Cornell Medical College, located in New York City) when students wrote a traditional argumentative 
research-based paper versus a graphic novel scenario using online graphics software. Thus the study 
investigated both modes of learning (blended online learning versus completely face-to-face instruction) and 
rhetorical modes (written essay versus graphic narrative). 


4. METHODOLOGY 

Exploratory investigations to test the viability of using an online graphics software program for medical 
ethics training among premedical students began in March of 2014 in a four- week module on Islamic 
bioethics within a sixteen-week history of medicine course entitled Perspectives on Islamic Medicine offered 
at WCMC-Q. All nine students were instructed to choose as a final project a controversial medical ethics 
question (scenario or case) in Islamic medical ethics, a growing field of scholarship which investigates the 
intersections among customary Muslim modes of thought and behavior (al ‘urf), shariah law, contemporary 
juristic reasoning ( ijtihad ), and novel bioscientific innovations such as stem cells, animal cloning, artificial 
life support, assisted suicide. Artificial Reproductive Technologies (ART) and organ transplantation. After 
reviewing peer-reviewed literature and formulating arguments and counterarguments on a specific ethics 
question, students translated their dilemma into a narrative using online comics software Pixton 
(www.pixton.com). These scenarios were required to be evidence-based, i.e. students were asked to imagine 
a real-life medical ethics encounter that could occur in an Islamic context, either involving Muslim patients 
and Muslim doctors or taking place in a nation following shariah law. 

In Spring, 2015, this assignment-broadened to include general and Western medical ethics-was repeated 
with 22 premedical students who were divided equally into control and study groups, one writing an 
argumentative paper and the other a graphic narrative. The objective was to compare differences in 
self-reported learning about medical ethics between the two different types of final project using an end of 
semester 10 question self-report questionnaire. The primary bases of evidence (3 peer reviewed sources were 
required both to write the argumentative paper and to construct the graphic narrative scenario) were drawn 
from the student’s personal experience, peer-reviewed articles and book chapters from the course readings, 
and research obtained from the Georgetown SFS in Qatar’s Islamic Medical and Scientific Ethics (IMSE) 
Database and related reference collection. 
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5. DISCUSSION AND RESULTS 

Pixton was first introduced into the author’s Spring, 2014 course. Two out of the 9 students encountered 
serious difficulties in using Pixton.com and subsequently used the accounts and assistance of other students 
in the class in order to complete the assignment. Follow-up revealed that these students were from 
first-generation college-goer families and did not possess full computer literacy. Standard end of semester 
feedback evaluation forms indicated that the software was successful in teaching the learning outcomes of 
Islamic medicine and Islamic medical ethics, and made the experience of learning more enjoyable. Thus the 
author proceeded to a more formalized and randomized pilot study in the Spring, 2015 semester. 

In a Spring semester 2015 medical ethics module, 22 students participated in the following activities: 
course readings from Beauchamp and Childress’s Principles of Biomedical Ethics , 7 th edition (2013), class 
discussions, role-playing, and online YouTube videos (documentaries of the Terri Schiavo case and AIDS 
activism). During the second week of the module, students were randomly assigned to a control group who 
were instructed to write an argumentative / persuasive essay and an experimental group who used Pixton.com 
to create a graphic narrative. Due to time limitations, instructors assisted students in choosing appropriate 
peer reviewed medical ethics sources to complete their final projects. Sources included standard bioethics 
journals, as well as the AMA’s Virtual Mentor American Medical Association Journal of Ethics 
(http://virtuahnentor.ama-assn.org/). Students who chose a topic in Islamic medical ethics made use of the 
Islamic Medical and Scientific Ethics (IMSE) Database at Georgetown School of Foreign Service in Qatar 
(http://imse.library.georgetown.edu/), the first database in the world dedicated to Islamic bioethics. 

After the completion of the final projects, students were administered a 10-question questionnaire 
containing 5 free response questions, and 5 seven-point Likert Scale questions with a scale ranging from 
“Entirely Disagree” to “Entirely Agree.” Seventeen students completed the questionnaire. The mean score 
for the question “Enough training and support was provided to use Pixton software” was 5.8 indicating that 
the majority of students did not encounter difficulties in using the software. However, 5 out of 1 1 students 
reported problems writing the graphic narrative; 3 problems were related to creating a story and difficulties 
with narratives in general and relating the story to medical ethics, and 2 problems related to technical issues 
with Pixton. For the follow-up study described below, the author plans to provide more detailed in-class 
training using Pixton with a hands-on workshop and online tutorials. Also, although a medically related 
graphic novel was assigned as course reading, little time was spent in class doing an in-depth analysis of the 
graphical and narrative elements. The mean response for the graphic narrative writers for the question “the 
final project was helpful in learning about medical ethics” was 5.0, while for the argumentative paper 
students the mean response was 4.7. In an unpaired t-test at the 95% confidence level, the two-tailed P value 
was 0.6876, demonstrating that in student self-report, there was no statistically significant difference in 
graphic narratives and argumentative papers in learning about medical ethics. This preliminary data on a 
small sample may indicate that the two rhetorical modes are equally effective in teaching medical ethics 
concepts. Thus the graphical narrative could be added to the range of pedagogical approaches in teaching 
medical ethics, such as the traditional argumentative paper. 

From the observational notes of the instructors, the observors noted that most of the students, except for 
the few encountering technical difficulties, enjoyed the experience of using Pixton.com since it was a unique 
and new approach to learning that they had not encountered in high school. This confirms a growing body of 
literature in serious gaming and edutainment that adding both an online component and entertainment aspects 
to learning (‘making learning fun’) can positively impact student motivation for learning. 

5.1 Follow up Study 

The author is planning to carry out a more rigorous experimental design (a randomized controlled trial) by 
comparing the pre- and post-test scores on a validated multiple choice exam measuring knowledge of basic 
medical ethics concept related to the human dimension of medical ethics. The test will measure student 
mastery of factual medical ethics decisions which relate to interpersonal relations, affective responses, the 
narrative structure of ethics cases, and human motivation. This perspective contrasts with other modalities of 
medical ethics teaching, such as policy-based approaches (following and applying rules) or legal medicine 
(complying with state and federal statutes). 
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In the follow-up research, the Jefferson Scale of Empathy-S-a widely used psychometric test that 
measures medical student empathy, will be additionally administered as a pre- and post-test to elucidate 
differences in empathy change when comparing learning through a traditional argumentative essay versus the 
graphic novel. The Jefferson Scale of Empathy-Student version (JSE-S), is a 20 question self-administered 
7-point Likert scale scored test available from the Center for Research in Medical Education and Health Care 
at Thomas Jefferson University in Philadelphia, Pennsylvania, USA. A preliminary reliability study of the 
JSE-S will first be carried out on a target population in Qatar. Reliability of questions will be measured 
through Alpha - item-deleted, item-total correlation, and inter-item correlation. 


6. CONCLUSION 

Ethnologists have yet to uncover a human society which does not include some form of music or storytelling, 
including cosmogonies, myths, memorializations of past events, heroes and deities, etc. This suggests that 
narrative may be genetically encoded and fundamental to human cognitive functioning, stretching to perhaps 
neurolinguistic structures of the human brain. Further investigation into the microstructures of the human 
brain at the neuronal level, such as The NIH BRAIN Initiative (Brain Research Through Advancing 
Innovative Neurotechnologies) started in 2013 or the related European Union collaboration called the Human 
Brain Project (which collectively will map all the neurons of the human brain, and create informatics models) 
will undoubtedly reveal how narrative and graphic narrative structures specifically organize, affect, and 
impact the phenomenological properties we call memory, experience, emotion and cognition. In the 
meantime, various qualitative (narratology, discourse analysis, postmodern philosophy. Philosophy of Mind) 
and quantitative (randomized controlled trials) research methods can perhaps elucidate how graphic 
narrative, and specifically electronically-enabled experiential learning in the creation of graphic narrative, 
can impact learning. 
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ABSTRACT 

The results of several studies conducted to analyze the quantitative and qualitative use of learning technologies in Higher 
Education in Portugal showed that, in general, these technologies are not used systematically and effectively and 
e-learning platforms tend to be relegated to repositories of contents rather than as full-fledged tools actively enabling and 
promoting learning. Several gaps were also identified at the management, pedagogical and technical levels. Following 
best-practice in this domain, a holistic intervention including these three levels is crucial to achieve positive results. 
Therefore, following these studies, a set of guidelines and recommendations was created to raise the level of technology 
adoption in higher education and improve the educational performance of each institution by raising teaching/learning 
levels so that they are in accordance with the Bologna paradigm. This way it is possible to have a systematic approach, 
expanding the intervention to the entire institutional campus and resulting in an effective learning platform usage by 
teachers and students. 
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1. INTRODUCTION 

The Web 2.0 advent resulted in the creation of structured online platforms where everyone could publish 
their own content, therefore opening the doors of internet publishing to the general public. Learning 
Management System (LMS), or e-leaming platforms, are specialized online platforms that support e-learning, 
that is, online learning and training through content and communication sharing. From an early moment the 
LMS Moodle was one of the favorite platforms for two reasons: it was as free and it had all the versatility of 
other e-learning platforms but still keeping its concept and usage simple. Nowadays Moodle has millions of 
users and thousands of schools use it in a daily basis. 

To understand the actual status of the use of learning technologies in the Portuguese Higher Education a 
case study was conducted, structured in three basic stages, as shown in Figure 1 : 

- Study for analysis of surrounding context; 

- Intervention to specify the parameters of the research and its implementation; 

- Assessment and Reporting, for data collection and formulation of conclusions. 
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Figure 1. Methodological reseach scheme. 
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The study began with a bibliographical research on existing projects and initiatives. Then, to collect more 
specific and updated results, three analyses of the use of the Moodle platform were done, allowing the 
identification of gaps which needed solution. Based on the conclusions of those analyses a set of 
recommendations was designed to take full advantage of the use of an e-learning platform in a particular 
institution, without losing content value and maintaining a balanced experience for all stakeholders. Figure 2 
depicts in more detail each of the sub-steps. This set of rules or recommendations was also influenced by the 
application of the Bologna paradigm (European Union, 2006), so that the use of this platform is a starting 
point for the student’s exploration of many subject related with his/her course. 
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Figure 2. Steps towards the use of e-learnig. 

The purpose of management intervention is to assure the creation of the necessary institutional conditions 
for the effective adoption of technology in teaching. This would lead to a standardization of all departments 
in the institution, contributing to the improvement of results at a teaching/learning level in the context of 
Bologna. 

Management intervention should address political-strategic decisions. The role of the researcher includes 
the management persuasion about these studies reliability and adoption of the proposals. A proposal to 
include teacher’s department in this initiative was presented with the intention of improving teacher’s tasks 
through the platform usage. CUs approval rates would, therefore, be also improved. The proposal at 
technology level embraces the personalization, usability and customizing the platform features and tools to 
the users. 

This implies the development of 4 or 5 software modules to be integrated in the platform. The aim of the 
intervention at this level is to upgrade the platform into a user-friendly tool for teachers and students so that it 
is more profitable to the teaching/learning process in the context of Bologna. E-learning platform success in 
the user’s community is, this way, enhanced. 

The analysis of the implementation of all proposals will then check whether your application and action 
taken at the institution has made recommendations to the Bologna process of teaching/learning in Fhgher 
Education leading to a positive influence on the current practices of the institution. 


2. STUDY 

The initial part of the research process was divided in two main stages: 

• Literature Review - Starting point of that work, allowed to ascertain the key areas. An analysis of the 
Bologna declaration and the associated paradigm, as well as their influences on the Portuguese higher 
education has been made. We also studied the technologies to support the teaching / learning process. 
We also analyzed the adoption of models of technology and adapted to the particular situation of use 
as educational support in higher education. 
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• Analysis of Current Situation - This step allowed to take knowledge of the current situation of 
adoption and use of technology-based educational tools in higher education, focusing on the use of 
Learning Management Systems. The reference model was UTAUT - User Acceptance Model of 
Technology Use. The use of an e-learning platform (Moodle) in three different scopes (always in the 
scope of higher education and, more specifically, engineering) was analyzed (Marques et al, 2010a) 
(Marques et al, 2010b) (Marques et al, 2011). 

Data was collected in three consecutive academics years. The use of the platform as a content repository 
is far lower than it would be expectable (from 30% to 60% of the disciplines use it as such, depending on the 
department) but still high when compared with the number of teachers that effectively use the e-learning 
platform as such, that is, exploiting the existing tools like assignments, quizzes, discussion forums, chat, etc. 
In fact, and even if the use of the e-learning platform by the teachers slightly increased during the three years, 
in the end the level of usage was still between 10 and 25%. There is some unbalance between departments in 
the same school. Computer Science and IT related departments have a higher adoption ratio, while in other 
departments the use of the platform is hardly relevant with values from 2 to 6%. Some reported contributing 
factors for this low level of usage were the lack of time to learn how to use the platform, operational 
difficulties and lack of knowledge of the platform potential. 


3. INTERVENTION 

The next two steps in the research process were the following: 

• Proposal of Performance Model - As a result of the above sub-steps in the intervention stage a 
performance model was proposed to address deficiencies identified in the previous substep. This 
model proposes: 

o In terms of management, political and strategic decisions (see details on section 3.1). 
o In terms of teachers, actions such as training/ issemination, coaching and persuasion (see 
details on section 3.2). 

o At the technological level customization, improved usability and automation tools (see 
details on section 3.3). 

• Model Implementation - This step corresponds to the performance in the field by applying the 
proposals set out above. 

The purpose of management intervention is to assure the creation of the necessary institutional conditions 
for the effective adoption of technology in teaching. This leads to the systemization of use, contributing to 
the general improvement of results at a teaching/learning level. The purpose of the intervention at 
pedagogical level is to create the necessary conditions to help teachers perform their tasks, with the 
e-learning platform, subsequently improving those tasks. Finally, the technologic intervention is the ground 
base for the creation of modules and tools that both students and teachers can use to have a friendlier and 
more intuitive platform. 

3.1 Management Intervention 

The intervention at the management level involves political decisions in order to regulate and organize the 
institution so that the intervention has an entire campus scale, involving the largest majority of students and 
teachers sharing best practice so that large gains are achieved in each discipline’s success rate. The ultimate 
goal is then to raise the learning technology usage to a level that enables the improvement of the quality of 
the organization in the teaching/learning processes. But, at this level, the researcher’s role can only be to 
persuade the governing body to adopt its proposals. 

• G1 proposal: Directors and managers should play an important role in spreading the adoption of 
technology usage in the teaching/learning process. The organization’s mission statement should 
clearly include the use of technology in the teaching/learning processes and set its goals: 
o Promoting the participation of primary stakeholders (teachers and students); 
o Raising their interest in the adoption of learning technologies. 
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• G2 proposal: There must be a clear leadership for a fully efficient technology adoption. The 
appointed leader should be responsible for strategy formulation and plan implementation: 

o Defining a 3-5 years strategy plan including historical data, trends and projection analysis; 

o Assessing the effectiveness and efficiency of resources (technology and human) to answer 

students and teacher’s needs; 

o Designing operational plans and regular activity reports with a clear identification of the 
strengths and weaknesses to improve the quality of service; 
o Providing support to the stakeholders and ensuring the availability of resources; 
o Verifying measures taken to ensure data integrity. 

• G3 proposal: The use of the e-learning platform by all curricular units (CUs) should be mandatory. 

o Requesting the mandatory use of the platform so that teachers may benefit from it (by increased 
use of the embedded learning tools) and students have the opportunity to access more diversified 
sources of content and communication with colleagues and teachers. This way CUs can become 
more interactive experiences. 

• G4 proposal: Incentives should be provided to encourage teachers to fully use the available 
platform. 

o Promoting the report and sharing of best -practice. 

• G5 proposal: The adoption of the technology should be frequently monitored. At a teaching/learning 
level the monitoring and evaluation of the adoption of the technology should be performed at an 
external and internal level. This process involves: 

o Performing periodic evaluations along with process and validating results in order to measure 
the attainment of goals and the progress of the strategy plan; 

• G6 proposal: Curricular unit planning should describe the use of learning technologies: 
o Planning in a clear and concise way; 

o Providing suitable technology and tools; 

o Including frequent interaction of students between themselves and with teachers; 
o Accommodating different learning styles; 
o Allowing students to have unrestricted access. 

3.2 Pedagogical Intervention 

This set of proposals take on the intervention at the pedagogical level, focusing on the teacher’s role and 
improving their pedagogical performance through the use of the e-learning platform, through training 
assessment. 

• PI proposal: Promote the use of e-learning technologies as a means of communication between 
teachers: 

o Exchanging information and best-practice between different UCs through synchronous and 
asynchronous tools embedded in the e-learning platform could represent substantial 
improvements of the pedagogical practice. 

• P2 proposal: Promote the use of technologies as a means of communication between students and 
also between students and teachers: 

o Using existing communication tools embedded in the e-learning platform should be promoted 
among students once it will actively facilitate relations between them and also students with the 
teachers. As a result a more close and interactive guidance with optimal learning experience will 
be achieved. 

• P3 proposal: Providing technical support to platform users. 

o Compiling teachers and students issues and providing adequate answers and solutions (for 
instance as Frequently Asked Questions - FAQs), easily accessible on the platform in order to 
help other users with the same or similar questions. 

• P4 proposal: Providing pedagogic support on the e-learning platform usage. 

o Making tutorials and guides available to help teachers organize and present information using 
the best pedagogic practices so optimal quality information is provided to students. 
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• P5 proposal: Provide teachers training support in order to help them use the platform. 

o Providing continuous training to teachers in order to take full advantage of the platform 
potential. 

3.3 Technological Intervention 

The proposal at technology level embraces the personalization, usability and customization of the platform 
features and tools to the needs and abilities of the users. This implies the design and development of 
applications (plug-ins) to be integrated in the platform so that it is upgraded into a user-friendly tool for 
teachers and students, raising the user experience for them. 

• T1 proposal: Summary plug-in 

o Keeping record of each lesson’s contents is a teacher common task, and it could be beneficial if 
students could access that record directly from the e-learning platform. 

• T2 proposal: Collaboration plug-in 

o Sharing files and information help the cooperation between workgroup members. Shared edition, 
versioning and other tools add value to the platform and improve student-student, 
teacher-student and teacher-teacher collaboration. 

• T3 proposal: Integrated social networking. 

o Developing modules based on social networks like wikis and chats is a way of both promoting 
the platform usage and adding to the information availability. 

• T4 proposal: Overall improvements. 

o Improving system security and stability, infrastructure upgrading, integrating with other tools 
and reinforcing tech support. 


4. CONCLUSIONS 

This study has proven that, in spite of all the advantages of using e-learning platforms we are still a long way 
from taking full advantage of them. We have compiled a set of carefully selected proposals that aim to make 
the user experience uniform throughout the whole institution and add value to the platform and its tools but 
also easing teachers’ work with it. 

By following these rules the target institution is setting a path towards excellence, keeping a bond with its 
students and adding value to its work, setting the foundations to a promising future where it can be a 
reference to others. 
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ABSTRACT 

Innovation in Internet of Things (IoT) requires more than just creation of technology and use of cloud computing or big 
data platforms. It requires accelerated commercialization or aptly called go-to-market processes. To successfully 
accelerate, companies need a new type of product development, the so-called validated learning process. Furthermore, the 
development of the IoT paradigm has advanced the research on Machine to Machine (M2M) communications and has 
enabled novel acceleration platforms. However, there is a need for converging currently decentralized cloud systems, 
general software for processing big data and IoT systems. The paper describes a cloud platform that will bring together 
several services and instruments that the companies usually rely on in their go-to-market process. The main contribution 
of this paper consists in the integration of several services in the platform. The proposed platform will allow companies to 
bring their products to market faster and in a more successful manner, will be accessed via the Internet and will host the 
tools and virtual environments needed for the go-to-market accelerated process. Finally, is presented the acceleration 
results of a Cloud IoT architecture for processing big data from M2M telemetry. 
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1. INTRODUCTION 

Cloud Computing represents a relatively new concept that refers to an integrated service offered as a whole 
application, which offers access to information and data storage without the user having to know the physical 
location and configuration of the systems providing these services. 

Additionally, cloud computing is a general term for anything that involves delivering services over the 
Internet. Research currently in progress is extending the advances of M2M to Internet of Things (IoT) and 
cloud computing, by developing highly innovative and scalable service platforms that enable secure-, smart- 
and partly-virtualised-services (Bonomi, 2012). 

Validated learning is the practice of effectively measuring the accuracy of assumptions and using the 
results of the validation to understand whether the assumption was correct and if so, continue onto the next 
test (Munch, 2012). If assumption was not accurate, then it will be decided whether the strategy, assumption 
or feature needs to be improved or to change direction. The concept of learning cycles in platforms for social 
learning (Shabalina, 2013) must also be mentioned. A learning cycle is a concept of how people learn from 
experience. A learning cycle will take a number of steps or phases, the latter being followed by first. 

The paper is structured as follows: Section 2 presents the literature review in the field of virtual 
collaboration spaces, describing learning cycles and Validated Learning. Section 3 describes the approach of 
existing training methods and presents the proposed acceleration platform for innovation in industry. Section 
4 describes a Cloud IoT architecture of a platform for processing big data from M2M telemetry. The 
conclusion summarizes the contributions of this paper. 
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2. VIRTUAL ACCELERATION SPACES 

Collaboration is essential in accelerating innovation, because this is where the need for tools that support the 
dedicated collaboration arises. Research topics in this area include support simulations for business model 
learning. A serious simulation will help the development of business model learning solution. This will be 
coupled with an engine that will analyze the results of new ICT product and expert system releases and will 
propose some adjustments. The real key to accelerating business is validated learning, such as the validation 
of methods for a business model (Blank, 2013). 

The adoption of a validated learning method requires new ways of measuring the accelerated innovation 
process. Experiments for validated learning are pilot services and tools can be developed iteratively. The aim 
of the technological infrastructure is to support the achievement of these experiments. The existing 
infrastructure allows a quick start for each new experiment, which will be used for developing the new 
infrastructure. 

2.1 Validated Learning 

The validated learning process represents the systematical search for matches between technology and 
market by validating the mechanics of a business model. Consequently, validated learning consists in 
iterating rapidly between experiments, data collection and informed decision making. Each company must 
install its own version of a validated learning process focused on the context of its business and industry to 
which it belongs. 

The validated learning process is not just about learning a skill. It is about learning whether assumptions 
about the business model are correct or not. Adoption of validated learning process requires new ways of 
measuring progress on acceleration. 

Therefore, the validated learning process is defined as a process that is learned by applying initial ideas 
and measuring them to validate the effect. Each test of an idea is a single iteration of a wider process 
consisting of several iterations, in which one learns something and then successive tests are applied. 
Furthermore, to perform quality evaluation for e-learning environments a method has been proposed to build 
the quality tree by selecting the quality characteristics from a list of common characteristics applicable to the 
whole e-learning experience (Militant, 2012). 

2.2 Learning Cycle 

Using the learning cycle methods, the marketing innovation can be carried out by different companies from a 
modular unit that can be used in different stages of the go-to-market cycle. Learning cycles are social 
learning platforms. New players are involved in a less formal context to promote creative thinking and 
innovation. 

Diverse perspectives, ideas, interests and local knowledge are taken into account in order to find 
alternative solutions that are optimal in comparison with the established approaches. The resulting solutions 
can be tested in small-scale pilot programs and patented (Siroker, 2013). If they succeed, new approaches can 
be adopted in the official management cycle. 

Learning styles include a number of theories that suggest systematic differences in natural or usual pattern 
of individuals for the acquisition and processing of information in learning situations. A basic concept is that 
individuals differ in the way they learn. 


3. PROPOSED CLOUD ACCELERATION PLATFORM FOR 
INNOVATION IN IOT 

To acquire knowledge and qualified persons, IT&C system developers rely heavily on consultants, advisors 
and other providers of domain specific knowledge and can provide training and coaching programs [5], Thus, 
they seek processes that ensure quality installation and maintenance services at limited costs. Market 
requirements are constantly growing and it is a challenge for providers to accelerate knowledge, but IT &C 
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technologies can represent a major role in enabling industrial stakeholders to expand their businesses and 
innovate. 

The several services that will be integrated in the platform are: awareness creation services, trainings, 
seminars, online courses, coaching and mentoring services, software tools that support specific parts of the 
acceleration process, services that provide access to data needed to speed up go-to-market, services to look 
for partners, project management tools (Figure 1). 


Cloud platform 

Management Services Technical Services Service Providers 


Services to look for partners Awareness creation services Seminars 


Software tools that support 

Project management tools specific parts of the acceleration Online courses 

process 


Services that provide access to 

data needed to speed up go-to- Mentoring services Coaching services 

market 


Trainings 


Figure 1 . Services integrated in the platform 


We will briefly describe each of them: 

Service to look for partners is a management service that lets the users take a hands-on approach to a 
wide range of activities related to getting support, building projects and finding opportunities. 

Project management tools is a service that offers support for task management, resource allocation, 
tracking, Gantt charts, and much more. 

Services that provide access to data needed to speed up go-to-market are search based services. With 
these services SEO optimization on start-ups’ websites can be achieved and in this way the acceleration of 
go-to-market process can be obtained. 

Awareness creation services are the most effective and least expensive channels for online advertising 
and building a positive reputation. 

Software tools that support specific parts of the acceleration process provide a scientific approach to 
creating and managing start-ups and getting a desired product in the customers’ hands faster. 

Mentoring service helps the users to refine their business strategy, map out the direction that their 
business should be taking and assist in setting goals for their business to succeed. 

Seminars are meetings in which people can learn about a topic. They are usually interactive sessions 
where the participants engage in discussions about the delineated topic. 

Online course is an electronic learning method. The platform for online courses provides functions such 
as: controlled access to educational materials, systematic consultation or searching large volumes of text and 
multimedia content, creating content through embedded HTML editors, editing tests and tutorials, editing 
glossaries / dictionaries, importing and exporting of educational content in popular formats such as MS 
PowerPoint files, MS Word, HTML, PDF, RTF or pictures and movies; creating archives / directories of the 
whole resources. 

Coaching service is distinct from other forms of training because it focuses on the method of learning. 
Business coaching includes principles from sports coaching such as teamwork, personal excellence, and 
"going for the goal". A business coach focuses on helping an individual “learn what it takes” for him or her 
to improve existing capabilities, set meaningful goals, and be accountable for his or her results. A coach 
helps an individual understand and eliminate barriers to more effective performance. 

Trainings are organized activities aimed at imparting information and/or instructions to improve the 
recipient's performance or to help him or her attain a required level of knowledge or skill. 
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4. RESULTS FOR M2M TELEMETRY AND CLOUD IOT 
ARCHITECTURE 

In this section we present the acceleration results for an IoT architecture composed by a M2M telemetry 
system and cloud web servers (Figure 2). 



Figure 2. Architecture of the M2M telemetry system 

The addVANTAGE Pro software (Adcon, 2014), which is based on a client/server architecture, collects 
data from one or several Telemetry Gateways (receivers) and makes it available for viewing or for 
specialized analysis. The server is that part of the software where all the actual processing takes place. It is 
responsible for downloading data from the Telemetry Bridge, storing data into the database, starting and 
stopping extensions, and servicing clients as they connect. The software and telemetry devices work together 
to form the telemetry system, which can be defined as a system that allows the user to: 

• measure certain parameters over a predefined area; 

• send those parameters over relatively large distances to a central point; 

• process the parameters as needed for various applications such as agriculture, meteorology, 
irrigation control, water management, and environmental analysis. 

The electrically converted parameters are first stored in the memory of a remote telemetry unit (RTU). An 
RTU has its own intelligence in the form of a built-in microcontroller, which periodically performs several 
tasks, such as: interrogating the sensors, storing the measured data, checking the radio channel, checking the 
local battery status, and so forth. It is part of a remote station, which consists of the RTU, its assembly parts, 
and its sensors. The RTU is equipped with a radio module or a GSM modem, which allows for real-time 
wireless communication with a base station. 

The base station consists of a Telemetry Gateway (or receiver) and user PC. The Gateway acts as a 
network controller — at regular intervals (typically 15 minutes, but this can be changed) it requests data via 
radio or modem from the RTUs in the network. 

The receiver stores the incoming data in its memory, thus allowing the receiver to supervise a large 
number of RTUs and keep their data for a period of time without the need to download the data to the PC. 
The number of controlled RTUs depends on the receiver type, and some receiver models can handle over 
1000 units. 

Cloud IoT is represented by SlapOS, a decentralized Cloud Computing system. It can automate the 
deployment and configuration of applications in a heterogeneous environment. SlapOS supports IaaS, PaaS 
and SaaS applications (Saad, 2014). 

Big Data is typically considered to be a data collection that has grown so large it can’t be effectively or 
affordably managed (or exploited) using conventional data management tools: classic relational database 
management systems (RDBMS) or conventional search engines, depending on the task at hand. Today, 
classic RDBMS are complemented by a rich set of alternative DMS specifically designed to handle the 
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volume, variety, velocity and variability of Big Data collections (the so-called “4Vs” of Big Data). These 
DMS include NoSQL, NewSQL and Search-based systems. 


5. CONCLUSIONS 

To successfully accelerate, companies need a new type of product development, the so called validated 
learning process. A validated learning process systematically searches the fit between technology and market, 
by validating the mechanics of a business model. 

In this paper we describe how acceleration can be performed using a cloud computing platform, validated 
learning processes and learning cycles for innovation in IoT. Validated learning systematically searches for 
matches between technology and market by validating the mechanics of a business model and consists in 
iterating rapidly between experiments, data collection and informed decision making. 

Learning cycles are social learning platforms. New players are involved in a less formal context to 
promote creative thinking and innovation. As future work we will take into account perspectives, ideas, 
interests and local knowledge in order to find alternative cloud computing solutions that are optimal in 
comparison with established approaches. 
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ABSTRACT 

The main purpose of this paper is to present the importance of Interactive Learning Objects (ILO) to improve the 
teaching-learning process by assuring a constant interaction among teachers and students, which in turn, allows students 
to be constantly supported by the teacher. The paper describes the ontology that defines the ILO available on the Internet. 
Besides, it shows the implementation of tests conducted on the virtual environment eTutor. Finally, it presents the 
assessment of the results based on the use of ILO in the subject Introduction to Hardware and Computing taught to 20 
students. 
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1. INTRODUCTION 

Thanks to the use of Technologies of Information and Communication (TIC) combined to the Internet, 
several resources have been providing great support to teachers in the teaching -learning process [6], Besides, 
time and location have been eliminated as obstacles in the educational field. 

Among the several technological resources, there are the Learning Objects (LO), which are units of 
learning that exist in order to support the teaching-learning process. Although creating LO is complex due to 
the absence of user friendly tools of creation [7], they can promote significant progress in the 
teaching-learning process through TIC [6], The Learning Objects (LO) help to create knowledge by 
incorporating features of interaction, adaptation and reutilization in several diferent contexts [24]. 

All tutoring contents tagged as Learning Objects (LO) should allow access and understanding by both: 
computers and users. However, they are generally described using language that can only be interpreted by 
human users. Therefore, to allow them to present meaning both to users and computers, [18] and [2] 
presented the Semantic Web. 

The Semantic Web enables users to create data archives on the Internet, as well as to write the rules to 
ensure their interoperability. One way of linking these data is through the data model called ontologies [8] 
[12] [13]. Besides the specification of concepts about domains [12] [20], the ontologies are also applied to 
improve the machine’s learning process and recovery of information [12] [13]. 

Several authors have pointed out the importance of using the Learning Objects (LO) in the educational 
context [4] [5] [6] [7] [24] [27]. The goal of this paper is to describe the Learning Objects (LO), which are an 
extension of the models described by [5], [26] e [27], using an ontology. 

With the resources mentioned above, teachers can follow each student’s learning curve and make 
appropriate interventions according to the individuals needs. The developed ILO were implemented on and 
assessed through the virtual environment eTutor. This paper assesses the ILO’s application during the 
Introduction to Hardware and Computing classes attended by 20 students from the Computing Science 
course. 

This paper is organized as follows: section 2 presents the use of ontologies for e-Learning; section 3 
mentions the definition of standards for ILO; section 4 shows the ontology that has been created; section 5 
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discusses and validates (with eTutor support) the ontology offered during classes of the Computing Science 
course; finally, section 6 shows the conclusion. 


2. E-LEARNING ONTOLOGY 

The ontologies are contemplated in several fields of Computing Science such as Web information recovery, 
through online products and services like Ontoweb [10], which is a smart system of agents that work towards 
the contextualized search on the accessed sources; SEAL [19], which enables the creation of semantic portals 
based on ontologies; and the OntoSearch [23], which facilitates the reutilization of the ontologies on the 
Semantic Web. 

The paper [9] presents the development of an ontology server that, by using LOM implements an 
ontology navigation service that dynamically develops HTML pages (including images and textual 
documentation), which shows the ontology’s hierarchy and uses HTML forms to permit users to edit and 
translate the ontology straight from the LOM’s model to the OntoLanguage. 

[14] focus his paper on cooperative learning, which utilizes one ontology to form groups focused on 
individual and group learning goals, roles and instructional learning events. 

Considering the models of domain, student, cooperation and pedagogical, [3] defines an ontology to build 
Interactive Learning Environments in order to answer the questions What, When and Where to study? 

To optimize search and the distance learning proccess, [11] has developed a tool based on agents, which 
is called Web Semantic Search (WebSS). Its main goal is to give conditions for recovering educational 
contents in the servers of a Learning Management System (LMS) that offers a highly respected and efficient 
tutoring platform also known as AulaNet. 

[17] applies ontologies in order to explicitly characterize the project and the learning objects, as well as 
the relationship among them. According to the authors, the application of ontologies may result in more 
efficient semi-automatic tools of extraction, which will , in turn, increase its reuse value. The authors also 
present a tool based on IMS-LD called Learning Object Context Ontology (LOCO). 

Some papers have been adopting ontologies to define learning objects, as presented in [1] which shows an 
ontology represented through the Web Ontology Language (OwL) [28] to demonstrate the Learning Design 
semantic (IMS-LD). Meanwhile, [25] describes the ontology as a Web based tool for the development of 
Learning Units. [25] mentions the use of ontologies to explicitly characterize learning projects, learning 
objects and the relationship among them. 

[22] employs an ontology to represent the Constructivist Learning by being able to identify the 
competencies and abilities that can be generated by the LO. Besides, they assure the reuse of the objects. 

Although the ILO presents the same concepts when compared to the traditionals LO [8], the ILO 
differenciate from the other proposals due to the fact that their main purpose is to promote interaction among 
students, teachers and environment. 


3. INTERACTIVE LEARNING OBJECTS 

Although the relationship of tutoring contents and LO with interaction has been described by [16] when 
presenting his metodology to build interactive LO, the author wasn’t concerned about structuring the 
construction of the LO by following standards such as LOM or SCORM. 

[21] demonstrates one tool of creation for LO that promotes for LO that promotes the interaction between 
the teacher and the LO, as well as among the student and the objects. However, the author doesn’t fulfill the 
demand presented in this paper, which is to assure the teacher is able to intercede depending on the current 
learning stage of the student. 

A LO must take into consideration several possibilities of interaction: between teacher and student, 
student and teacher, LO and student, and LO and teacher. [16] has pointed out the importance of the 
interaction processes in the cooperative learning models, which generates a higher demand for the creation of 
ILO. This interaction is supported through the use of educational tools such as chat, tips, alerts, messages and 
appendixes. 
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In order to fulfill the proposed interaction, the SCORM’s metadata standard was adapted. Thus, the 
compatibility with the traditionals LO was maintained. Among them, those from the interaction class are 
highlighted. They were created to specify all the educational context of the interaction, from the way it is 
perceived from the users point of view to how it is offered by the environment. 

As soon as an activity starts, the ILO offer options of interaction to the students, such as asking for tips or 
having a conversation with the teacher through chat. Resources such as attached files (videos, presentations 
and tutorials) are also available for students while they are using tips. 

The ILO assure one interaction that can be automatically turned on. For example, after a certain period of 
time without interacting with the ILO, the student has the option to receive a tip. At this point, the teacher is 
warned that the student has reached a certain period of time without interacting, which might indicate some 
difficulties to move forward. The alerts work based on a color code system that indicates where the student 
stands regarding his/her use of tips and time spent in the acitivity. The resources are implemented in a Virtual 
Learning Environment built with this purpose, eTutor. 


4. ILO S ONTOLOGY 

The ILO’s ontology was made possible by using the language OWL and by considering the principles of 
Semantic Web. Therefore, the intention was to facilitate the search mechanisms and to provide meaning to 
the data for the computers’ interpretation. 

Some important levels in the process of creation and validation of the ontologies are: vocabulary; 
hierarchy (taxonomy); context (level of application); Syntax Level; Architecture; Design and Structure [12]. 

The LearningObject class refers to the instances of the objects and it is related to the metadata classes: 
aggregations and content. The content class wasn’t described in this version of the ontology due to the fact 
that it belongs to the Learning Object’s tutoring content. 

The LearningObject class represents the instances of objects described along with its metadata. The 
Content class represents the composition of the LearningObject, which is not included in this version of the 
ontology. The Vocabulary class encompasses, among the Ontologies and Standards used as reference, those 
metadata that have fixed values composition and can not be altered beyond those already forecasted. The 
relationship among the classes follows this format: Metadata hasSubClass General. 

Each element of the ontology is described in its formality, containing: class name, URL Equivalent Class, 
Disjoint Class and Annotations. 

All the 55 elements of metadata were used following the ILO Standard and divided in Data Properties and 
Object Properties. The first one refers to the simple kind of elements that can be directly related to the 
metadata. Following, in this case, the model its cname metadata>Is. The second one fulfills the compound 
elements as indicated in the Aggregations class and having the structure has <name metadatas 

In the Properties model of Objects all the relationships like has <name metadata> have an inverse 
relationship identified as itscname metadata>Of. 

The profile used in the OWL Language was OWL-DL, which is an intermediate version as important as 
the descriptive logic. Therefore, the profile is less formal than OWL Full and it has followed the 
specifications of the entire domain without further problems. The current ontology is available at: 
<http://www.nogueiraluz.com.br/projetos/ILOvl.owl>. 

The ontology presented in the previous section was implemented at the eTutor’s ILO tool and tested by 
20 students. The tests were given during the Introduction to Hardware and Computing classes from the 
Computing Science course. Assessments, reading archives, questionnaires, appendixes and tips were among 
the ILO’s activities. 

Regarding the tests, initially the students were taught how to use the environment, as well as how to ask 
for support in case of an intervention. The presentation also included the students’ perspective during an 
activity and the options available such as: tips, appendixes and Chat. 

The teacher has put together an ILO in a questionnaire format containing 6 multiple choice questions. 
Each question contained 3 tips con figured accordingly. The tips could either pop up automatically, after a 1 
minute interval without the student’s interaction with eTutor, or could be asked by the student at any time. 

After having finished the activities, the students have received a questionnaire, which the main goal was 
to map the students’ opinion about the interaction with the teacher, follow up and the ILO. 
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5. DISCUSSION 

All participants answered 6 questions after having finished the activities. The questions were meant to assess 
the ILO. The percentage, as well as the questions are presented in Table 1 and the quantitative assessment 


and the percentage of answers found are shown too. 

Table 1. ILO Assessment questions 

Questions 

Yes No Does not apply 

1- Have you ever been in a Virtual Learning Environment before? 

2- Did you feel like the teacher was supporting you ? 

3- Did you ask for tips during activities? 

4- When needed, Did the teacher offer you help and activity follow up? 

85 

15% y Does not apply 

85% Does not apply 

45 

55% y Does not apply 

65% 5% 30% 

Questions 

Excellent Great Regular 

5- What is your opinion about interaction during activities? 

6- How do you rate eTutor’ s tool and the Interative Learning Objects? 

45% 30% 25% 

55% 30% 15% 


According to this assessment, 85% rated the ILO as Excellent and Great. Likewise, 75% of the students 
rated the interaction of activities as being positive. Only 15% of students had already used other LO in 
Virtual Environment (AVA) before, against 85% who had never accessed an AVA before. 

The students who already knew other AVAs, reported that they felt the teachers gave the needed support 
when using the ILO in the eTutor. They also rated the possibilities of interaction as being Excellent. In this 
group, 67% of students were giving the support they needed thanks to the help pointed out by the 
environment. 


6. CONCLUSION 

The demand for new tools of support to the teaching-learning proccess is evident and has been growing 
steadily. This paper presented the ILO, which are intended to improve the monitoring of student’s learning 
curve in order to detect their difficulties and apply the solutions accordingly, individually and immediatly. In 
addition, the teacher can plan the content to be taught according to the students’ needs. 

Consequently, this paper has defined the ILO ontology that permits the interaction among all participants: 
teacher and student; student and environment; and environment and teacher. The above mentioned ontology 
works according to the SCORM standard, as well as to the Semantic Web principles. As a result, it 
encourages ILO’s creation, padronization and sharing. 

eTutor was extremely important during the ILO’s assessment, once it permited not only the ILO’s 
assessment but also its testings. All the results were positive. 

For further investigation and future research, it is highly recommended not only to map the ILO 
ontology’s main classes: Content and LearningObject, but also to apply assessments with the coolaboration 
of professors from different fields of study. 
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ABSTRACT 

E-leaming in teaching and learning considered as the easy way to use information and communication technology by 
using of the internet. With the support of E-leaming higher education can be delivered anywhere and at any time. 
Although E-Learning is very importance in Libyan higher education. It’s implementation is facing many challenges in the 
Libyan universities. This paper focuses on the implementation of E-leaming in Libyan Universities. The main aim of this 
paper is to discover the implementation of E-learning in Libyan higher education and to identify the factors affecting the 
use of its technology. The research findings and recommendations will benefit the Libyan policymakers and the 
stakeholders. 
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1. INTRODUCTION 

During the last three decades, researchers have started to pay extensive attention to learning styles such as 
those working in the field of computer science who have adopted the idea of using learning styles in 
education, one of these ideas is E- learning in teaching and learning which can be defined as information 
presented and learned by using a network or Internet. 

E-learning used in colleges and universities to make learning more accessible and efficient. It gives 
students the chance to learn at anytime, anywhere, and at their own. Accordingly, e-learning improves the 
learning process by offering a different way of delivering education. This is does not prevent the fact that 
some People find it easier to understand a new concept by reading a textbook or verbal explanation. 

Despite the importance of E-Learning in higher education (HE), in Libya, a number of challenges still 
face the implementation of E-learning in Libyan universities such as ICT infrastructure; Leadership; Culture; 
and E-learning strategy. 

In order to achieve effective adoption and implementation of E-learning technology and to gain the 
potential advantages from employing this technology, this research will investigate the potential benefits of 
using e-learning in Libyan Higher Education and identify the factors affecting the use of E-learning 
technology, to make recommendations regarding viability and potential implementation. Recommendations 
of this study will benefit the teachers and learners as well as the policymakers who are planning and 
designing the Libyan HE system. 


2. LITERATURE REVIEW 

E-learning has started in many developing countries in order to convene the increase of the requirement for 
education and meet the increasing decline of trained teachers, (UNESCO, 2006). 
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2.1 E-learning 

There is a wide range of definitions for E-learning. It can be defined as learning facilitated and supported 
through the use of information and communications technology. Lahwal et al., (2009) clarified that 
E-learning is a kind of learning supported by Information and Communication Technologies (ICT), it is an 
interactive learning in which the learning content is available online and offers feedback to the student’s 
learning activities. Khan (2001) stated that E-learning can be thought of as anything that includes technology 
with interactivity to support learning, training and communication between groups and between individuals. 
It included web-based learning (WBL); internet -based training (IBT); advanced distributing learning (ADL); 
and online learning (OL). 

E-learning also can be easily operated for large groups of students such as online each group for instance 
can be in different place. Since in some instances of E-learning tools and materials, the lecturer and the 
learner can modify the learning materials to their own needs and suitability, students have more control over 
the learning process and have the possibility to better understand the material leading to a faster learning 
compared to instructor-led training. 

2.1.1 E-learning Challenges 

E-learning can be a sensible example of useful use of technology in life and this effective learning culture 
may meet the growing need of flexibility in facilitating education to the population. However, many 
challenges facing E-learning shows its growth in the world, different factors such as technical, economy, 
political and cultural conditions are the challenges to implement E-learning in different countries. 

Kenney et.al, (2005) clarified that E-learning was seen as a chance to cut costs and make savings in space 
and human resources, which are unlikely to be realized in the short term and development of E-learning by 
the government through funding allocations. In developing countries, it is difficult to provide different 
facilities and opportunities required to establish E-learning environments. On the other hand, some 
researchers believe that E-learning is not an appropriate way of teaching. For instance, Lytras and Pouloudi 
(2001) stated that it is a waste of time because E-learning courses often fail to support value of creating 
processes both for teachers and learner and their recognition requires huge efforts. 

Bleimann (2004) added that isolation by no social contact, missing part of the informal social 
communication and face to face contact of traditional classroom training, also, it may cost more to develop; it 
needs new skills in those who create course content. 

Sims and Sims (1995) stated that many challenges facing higher education institutions that increase the 
pressure on them, e.g. retaining enrolment, providing adequate resources, sustaining quality, raising funding, 
and strengthening the curriculum. Rhema and Miliszewska (2010) clarified that a number of technological 
challenges faces Libya, such as lacks in the required technological infrastructure, while availability of 
computer laboratories are available in most Libyan higher institutions, the network facilities is lack and 
makes a serious limitations on Internet access. Also, the limitation of using the educational software within 
institutions, and the technical support is almost unavailable in Libya. 

2.1.2 E-Learning in Libyan Higher Education 

According to Rhema and Miliszewska (2010) the implementation of E-learning and the use of ICTs in Libya 
are still in an early stage similar to in most developing countries. Although Libyan Universities still use the 
old method of education that depends on face-to-face communications between students and teachers in, and 
outside of, classroom, in campus. Some Libyan universities, such as Tripoli University, Benghazi University, 
and Academy of Postgraduate Studies and Economic Research, have the basic ICT infrastructure (such as 
computers, Internet access, and a local area network). 

Kenan et al., (2011) clarified the barriers of implementing e-learning in Libyan HEIs as E-learning in 
Libyan institutes is in early years, and still there are many challenges that affect the adoption of E-learning 
that can be categorised as: Technological, mismanagement. Cultural and other issues. They provided a set of 
recommendations for students and instructors to improve e-learning in Libyan HE institutions as follows: 

• Improve students’ computer skills before enrolling E-learning course; 

• The content of the e-learning course should be designed to meet the National Qualifications Framework 
(NQF) and Libyan Qualifications Authorities (LQA) requirements. 

• The attitudes towards e-learning should be taken into consideration. 
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• Teachers should support learners and institutions should employ an adequate staff members. 

The authors provided recommendations for institutions as follows: 

• Senior managers should support the implementation of e-learning and allocate a fixed budget. 

• Instructors should be offered adequate training and course developers to be updated on changes to 
software and hardware. And tools should be kept up to date at all times. 

Some recommendations for governmental HE policies have been addressed by the authors as follows: 

• Share coordination and resource between different institutions and encourage the successes institutions, 
to share their success with other institutions. 

• Develop partnerships between government, and the private sector. 

3. METHODOLOGY 

A semi-structured, face-to-face interview technique used to collect the necessary data from the Libyan 
students and lecturers from Tripoli university faculty of engineering department of architecture in order to 
explore their opinions, as well as to give their advice about how to find solutions to the problems of 
implementing E-learning in Libyan higher education. Furthermore, to understand their opinions of current 
and traditional way of studding in Libyan higher education institutes. Also, researchers used one students’ 
study group to make an argument about some points related to using E-learning in higher education, and find 
out the problems and solutions to the problems. 


4. MAIN FINDINGS 

4.1 Summary of the Students’ Main Findings 

• Most of students are using software, social media and internet in their study as a kind of E-learning. 

• Almost all students believe that E-learning is an appropriate way of teaching and studding. 

• All students have chosen the all benefits addressed in the literature, the most benefits of using E-learning 
are because it provides a flexible learning and help built self- knowledge and self- confidence after that 
mange time, improve preservation and reduce cost. 

• Most of students have used E-learning in some courses. 

• All students believe that E-learning services in the Libyan higher education are not good. 

• Students have chosen many reasons of the limitation of using e-learning in Libyan institutes such as: 
Limitation of using E-learning in Universities; lack of ICT infrastructure; lack of personnel qualified; 
Limitation of using the educational software within institutions and resistance to change from teachers. 

• Students selected the all options of E-learning challenges collected from literature, the highest rate went 
to the lack of implementing e-learning by government and lack of public awareness about ICT, after 
that the limitation of basic infrastructural resources and people do not accept the concept of E-learning. 

• Most students are using social website to communicate with the University. 

• The majority of students preferred to use computer in presenting lectures. 

• Most of students prefer to use computer for doing homework. 

Other comments raised by students about the benefits of using E-learning are: 

• Enable people to teach anytime and anywhere and save more time. 

• Help student to do research for homework 

• Need computer to log in into university web site and don’t need to carry papers or documents 

4.2 Summary of the Lecturers’ Main Findings 

• Although Universities did not provide any kinds of E-learning, most lecturers are using computer, world 
wide web, and software such as PowerPoint to deliver lectures. 
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• E-learning can be used for many purposes: delivering courses, lectures and instructions, learning over 
distance, conferences, communications between lecturer and students. 

• There were computer labs for students, and internet access for teachers before the revolution of 
17/2/201 1, but because of the armed conflict, many things are broken. 

• E-learning in general are very useful way to teach and learn in all learning stages for many reasons 
summarised as follow: 

Most E-learning programs can be taken when needed; Spread and share knowledge and saves the 
students time, effort and cost; E-learning ensures that all learners receive a consistent message of the 
target learning; E-learning can accommodate the various learning styles to get the message more 
effectively by using audio, visuals, interactivity, and simulation; It's self-paced as most E-learning 
programs can be taken when needed; Easy way to find information, references, lessons etc., and 
Flexibility to join discussion areas at any hour, or meet with colleagues and instructors in chat rooms. 

• Libraries in Libyan Universities are still using the tradition way, however, the department of 
architecture provided an electronic library includes materials collected by lecturers and students. 

• The University web site, have got a registration in many electronic libraries and scientific journals for 
teachers and postgraduate students. 

• The University communicate with teachers and students through the University electronic gate web site. 

• communication between the department, teachers and students are by using facebook and e-mail 

• Barriers of implementing E-learning in Libyan Universities as follow: 

Weak infra structure (servers, internet reliability, lines expensive); Political issues; Administration; 
Lack in physical resources, including accommodation, computer and other ICT related resources, 
connectivity and access for those people with individual needs; and Shortage in information advice and 
guidance, to ensure that learners are on the correct learning programme at the right level, advice on 
progress into new learning programmes, monitoring, assessment and on-programme support. 

• Lack in learning management and use of management information in order to improve the quality of 
content and teaching. 

• Many points to improve and apply E-learning in Libyan higher education as follow: 

Provide a generous funding and lots of careful planning and expertise; Offer good management and 
administration; The right person in the right position; Improve ICT infrastructure; Connect with 
advanced resources over the world; Provide qualified staff and Improve students’ computer skills; 
Changes should be taken gradually in a correct manner; and Employ people who have skills. 

• The future of Libya will be excellent, because Libya have got many things could make it one of the best 
countries in the world. 


5. CONCLUSION AND RECOMMENDATIONS 

The findings of the study suggest some useful recommendations that could be used in achieving the aim of 
this study which is to analyse the opportunities for E-learning in Libyan Universities, in order to assess the 
viability and benefits of its introduction in present and future developments. A variety of points, suggestions 
and recommendations are highlighted in this section. The objective is not to create a definite strategy but to 
recommend some general principles. 

Recommendations for lecturers and students to improve E-learning in Libyan Universities as follows: 

• Improve the methods of teaching and learning and follow the world’s education market. 

• The content of the E-learning course should be designed to meet the National Qualifications Framework 
(NQF) and Libyan Qualifications Authorities (LQA) requirements. 

• Instructors should support learners and encourage them to use E-learning in their study. 
Recommendations for institutions as follows: 

• Improve learning management, including the management of the curriculum, staff and use of 
management information. 

• Senior managers should support the implementation of E-learning and allocate a fixed budget. 

• Instructors should be offered adequate training and course developers to be updated on changes to 
software and hardware. 
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• Tools should be kept up to date at all times. 

• Connect with advanced resources over the world. 

• Provide a qualified staff and changes to e-learning should be taken gradually in a correct manner. . 

• The attitudes towards E-learning should be taken into consideration. 

• Better training to students and staff, and better ongoing technical support from the system vendor 

• Provide appropriate training at different levels, 

• Make researches to gather data for future developments; 

• Libyan institutions have to accept E-learning and to fulfil the potential of ICT and to get quick 
technological development, Libyan institutions may follow a neighbouring countries’ experiences for 
teaching and learning. 

Recommendations for governmental higher education policies as follows: 

• Improve physical resources, including accommodation, computer and other ICT-related resources. 

• Provide a generous funding and lots of careful planning and expertise. 

• Improve ICT infrastructure. 

• Share coordination and resource between different institutions and encourage the successes institutions, 
to share their success with other institutions. 

• Increase partnerships between government, and the private sector. 

• Convince the community and the learners’ needs, 

• Integrate E-learning in the education system. 

• Libyan government should pay more attention and great effort to ensure the development and motivation 
towards e-learning and suitable responses. 

Finally, it can be summarising that students and lecturers indicated a positive attitude towards E-learning. 
E-learning in Libyan higher education institutions as in many developing nations is still remains an 
unexplored entity and further research is needed to discover and determine the types, and challenges of 
educational settings as associated to ICT and E-learning in higher education institutes in Libya. It is wished 
that the findings of the dissertation will be as a source of information for academics, and decision-makers 
involved the implementation of E-learning in Libya. Clearly indicate advantages, limitations and possible 
applications. 
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ABSTRACT 

Digital learning games are useful educational tools with high motivational potential. With the application of games for 
instruction there comes the need of acknowledging learning game experiences also in the context of educational 
assessment. Learning analytics provides new opportunities for supporting assessment in and of educational games. We 
give an overview of current learning analytics methods in this field and reflect on existing challenges. An approach of 
providing reusable software assets for interaction assessment and evaluation in games is presented. This is part of a 
broader initiative of making available advanced methodologies and tools for supporting applied game development. 


KEYWORDS 
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1. INTRODUCTION 

Applied games are games serving a primary purpose that goes beyond the aspect of pure entertainment. Most 
common in this field are digital educational/learning games, which represent an increasingly used e-learning 
technology. Their highly motivating character makes them effective educational tools for creating authentic 
learning tasks and meaningful, situated learning (de Freitas, 2013). With the application of educational games 
for instruction and learning there comes the need of accounting for learning game experiences also in the 
context of educational assessment. Conventional educational measures are not suitable in the context of 
educational games, since they are usually highly invasive and compromise flow (Van Eck, 2006). 

Learning analytics (LA) is considered as a key approach providing new opportunities for learning 
performance measurement, assessment, and improvement in and of applied games. Translating LA to 
learning games combines two major trends in e-learning and stimulates new research questions in both fields. 
After introducing the general notion of LA (section 2), this paper provides an overview of current analytics 
research and developments in applied games (section 3). Based on a discussion of existing challenges, in 
section 4 an asset-based approach is outlined that supports realizing and integrating LA in game technologies. 
This approach shall foster a broader uptake of applied games and LA by the game industry and facilitate their 
broader adoption in educational practice. 


2. BASIC NOTIONS OF LEARNING ANALYTICS 

LA is defined by the Society of Learning Analytics Research (http://www.solaresearch.org/) as “the 
measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of 
understanding and optimizing learning and the environments in which it occurs”. LA is used to empower 
learners, instructors, and educational institutions, as well as to support intelligent tutoring systems. LA 
consists in a multi-step, cyclical process of data collection and pre-processing, analytics and action, and 
post-processing (Chatti et al., 2012). Data collection and pre-processing involves the gathering of educational 
data from different learning tools or applications and preparing and translating it into an appropriate format. 
The analytics and action phase denotes the actual application of analytic methods (e.g. structure discovery, 
relationship mining etc. - for an overview see e.g. Baker & Siemens, 2014) to extract meaningful patterns 
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and information from the data and to make use of the obtained results, (e.g. visualization, feedback, 
recommendations, adaptation). Post-processing refers to the idea of continually improving analytics, by 
refining analytics methods, using new methods, including new data sources etc. Until now, a lot of work in 
LA has focused on researching the methods of data collection and analysis; only recently more intensive 
efforts on their application in educational practice are being made (Ferguson, 2012). 


3. LEARNING ANALYTICS IN APPLIED GAMES 

By recording user ( interactions on a micro level learning games produce large amounts of data that may be 
used for LA. In fact, all digital games use in-game mechanics to appropriately respond to gamers’ actions. 
These analyses, however, focus on assessing playing performance instead of learning (Baalsrud Hauge et al., 
2014). A crucial question is how to harness and make sense of game-based user data in an educationally 
relevant manner. The valorisation of analytics for learning game assessment has started only recently. By 
combining ideas from gaming analytics, web analytics and traditional LA it is possible to establish 
meaningful analytics for educational purposes. In general, two types of LA in applied games can be 
distinguished (Westera et al., 2014): In-game analytics (real-time) and post-game analytics (off-line). 

A post-game approach to LA refers to supporting summative learning outcome measurement or 
identifying general learning patterns, and - more rarely, though - to the evaluation of the game artefact. 
Serrano-Laguna et al. (2012, 2014) proposed a two-step generic approach for LA in educational games, 
which is applicable with any kind of different game. Generic traces are gathered from gameplay, including 
game traces (start, end. quit), phase changes (game chapters), input traces (mouse movements, clicks), and 
other meaningful variables (e.g. attempts or scores). These data give rise to reports with general and 
game-agnostic information, like the number of students who played a game, average playing time, game 
phases in which users stopped playing etc. Visually reported, this information provides initial useful 
information on how learners interacted with a game. In a second step, additional information may be 
extracted by letting teachers define assessment rules based on and combining generic game trace variables to 
obtain new information (e.g. setting maximum time thresholds). These rules are defined closely in line with 
each game to match the educational objectives. A more specific approach of realising post-game LA was 
presented by Westera et al. (2014), who have used correlational and regression analyses to investigate 
switching behaviour between game objects and activities as predictor for learning performance. 

In-game LA is designed into a game and usually serves two purposes: Providing teachers and learners 
analytics results as a basis for action (e.g. selection of educational resources, decision on additional support 
or learning tasks etc.), or realizing dynamic adaptation during game time. Such stealth or embedded 
assessment avoids disrupting game experience, since assessment is appropriately integrated in the game and 
carried out non-invasively (e.g. Bellotti et al., 2013; Snow et al., 2015). Stealth assessment is usually 
implemented by linking observable game behaviour with an underlying model of learning outcome, 
competence etc. and regularly updating the user model (e.g. based on Bayesian score models or 
Competence-based Knowledge Space Theory). In this way, learning can be monitored and fostered, for 
example by generating progress reports and selecting new game experiences (Shute et al., 2009). Continuous 
non-invasive assessment was implemented in the educational games developed in the ELEKTRA and 
80Days projects ( Kickmeier-Rust & Albert, 2010): In a nutshell, learner actions during a complex 
problem-solving situation are monitored and interpreted in run-time in terms of available and lacking skills, 
current competence and motivational state. The information coming from this assessment is used to trigger 
adaptive interventions from a menu of different intervention types and tailored to the individual’s current 
state and needs, in order to support and guide the learner in the game and learning task and to retain 
motivation (Kickmeier-Rust et al., 2011). Another example of in-game LA was presented by Baker et al. 
(2007), who realised skill assessment in an educational action game by using game events as evidence for 
users’ mathematical skills, and to analyse study gains in accuracy over time and in speed over time with 
learning curves. Such kind of approach proved useful for formative assessment in educational games and 
may also be used to inform re-design and improvement of intelligent tutoring systems. Ventura et al. (2014) 
outlined a method for assessing persistence in educational games based on the time that learners spent on 
unsolved problems; this information may be used to tune gameplay difficulty, feedback and hints. 
Stephenson et al. (2014) elaborated an automated detector of engaged behaviour in a simulation game. The 
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aim thereby was to identify and model which learner actions give evidence of user engagement and. in the 
end, are predictive for success in the game. An integration of an engagement detector in the game may enable 
reporting the results back to learners and teachers for reflection. 

To summarise , LA has started to spread into the field of learning games, which is in fact a development 
that strengthens the position of both types of technologies. Nevertheless, there are still several challenges 
ahead, some of which are discussed in the next section. 


4. EXISTING CHALLENGES AND AN ASSET-BASED SOLUTION 
APPROACH 

Despite the theoretical and empirical evidence for the potential of educational games, there is still some 
reluctance among teachers towards their broader take-up and use in educational practice; for example 
because assessment routines built-in in educational games are usually black boxes and not tangible for 
teachers (Serrano-Laguna et al., 2012). Likewise, leisure game developers are hesitant to enter the applied 
games market, due to the high effort involved in learning game development and the difficulties of measuring 
learning outcomes. There is a need to make available approaches for transparent and reliable assessment in 
educational games - approaches that are based on valid assessment models, that are easy to use and provide 
meaningful educational information, and that give game industry evidence on the quality of a game. To 
maximise the potential benefits of LA for learning with applied games, LA actually should be incorporated 
already in the design phase of an educational game (Baalsrud Hauge et al., 2014). A great challenge with LA 
in educational games is the wide variety of different games available, which complicates the development of 
generic analytics tools that are applicable independent of a concrete game. The technical challenges of 
providing technologies for data collection, aggregation, analysis, and visualisation are therefore high. 

The European project RAGE (http://rageproject.eu/) aims at making available interoperable 
methodologies and tools for supporting applied game development. An applied gaming ecosystem will 
provide centralised access to reusable software components (so-called ‘assets’). Among others, assets for 
user data analytics are developed that can be used to gather and analyse interaction data from games and to 
integrate automated assessment and adaptation mechanisms in them. Assets for on-line and non-invasive 
interaction assessment are implemented that enable game developers to easily specify domain models, data 
collection, and interpretation rules as a basis for competence -based and motivational assessment. These 
assessment routines will empower learning game environments by providing meaningful input for game 
balancing assets supporting adaptation to individuals’ competence and motivational states. 

Existing analytics approaches in educational games often focus on realising continuous assessment for 
automatic live adaptation. The application of LA should be strengthened with respect to feeding back the 
information on skills acquired and learning progress to learners and teachers in a suitable way. In RAGE LA 
results also feed into dashboards and visualisation assets for users. These may further leverage the 
educational value of LA and may potentially be transferred to educational actions outside a gaming context. 

In addition to supporting LA for measuring and reporting learning success and for dynamic adaptation, 
the asset-based approach of RAGE shall also serve assessing and improving an educational game itself. An 
asset is elaborated to enable in-game evaluation. This evaluation asset thus represents an instrument for 
continuous evaluation of the quality of learning games by providing insights to users’ perception of games 
and their progress towards game goals. This is done by translating log data into meaningful information 
about game quality, user experience, and learning based on pre-defined, configurable evaluation metrics. The 
asset will facilitate the use of analytics for game evaluation purposes and will advance evaluation methods of 
applied games towards a meaningful triangulation of different data sources. The RAGE technologies will be 
exemplified, tested, and evaluated in asset-based games (mobile and desktop implementations) targeting 
employability skills in the context of different application scenarios. 

There is still more work to do to fully exploit the potential that LA in educational games may bring to 
optimize learning experiences. Beyond addressing the need for advanced, controllable, interoperable, and 
flexible technologies that facilitate the integration of LA in games, it needs to be taken into account, in 
particular, that games may be part of multiple learning activities and tools that learners carry out and use in 
parallel and, potentially, on the same educational objective or domain (Miller et al., 2014). 
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ABSTRACT 

A myriad of cultural differences can take educators in a multitude of directions when planning intentional design 
strategies in an online course to become more inclusive of international student needs. The topic of cultural diversity is a 
complex discussion. Nevertheless, there are ways educators can begin to practice culturally inclusive strategies. Using 
Hall’s (1976) low-context communication style, this paper addresses two components of the online course, the syllabus 
and the learning management system, and offers suggestions for beginning the journey toward culturally inclusive 
pedagogical practices to address barriers in the curriculum for international students. 
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1. INTRODUCTION 

In 2013/2014, higher education in the United States saw a growth of international students by 8.1% for a total 
of 886,000 students. Specifically, undergraduate student enrollment increased by 9% and graduate student 
enrollment increased by 6%. A large number of international students are entering U.S. higher education 
from all over the world with the majority of enrollments from China, India, and South Korea (Open Doors 
2014). This continued growth places international students “as central players in intercultural exchange and 
diplomacy between nations” (Lee & Rice 2007, p.385). The opportunities for increased global participation 
in US higher education is due to the nature of online instruction. The ability to take courses from anywhere in 
the world from the comfort of one’s own home, office, or coffee house entices potential students. Online 
instruction has increased and there is no stopping its capabilities. 

New avenues for instruction require inclusive teaching philosophies in curriculum design and teaching 
practices (Hellsten & Prescott 2004). This paper briefly discusses cultural complexities but will focus on 
language proficiency as a major challenge for international students in online instruction. Following the 
discussion, recommendations are made to assist educators in leveraging the technology afforded by an LMS 
to adapt curriculum to be more inclusive of international students. 


2. CULTURAL FACTORS 

When we think of the diversity of our learners we consider a number of learner characteristics described by 
gender, age, socioeconomic status, ethnicity, physical and cognitive disabilities, and culture to name a few. 
As we look at culture, a myriad of lenses come to the forefront. In online courses, however, it is often 
difficult to determine the nationality or even the gender of our online students. Online students begin as 
raceless and genderless. As a result, it is imperative that we design our courses with sufficient cultural 
competence so that we might better prepare to meet the expectations of international students (Bista & Foster 
2011; Cartledge & Kourea 2008; Zhang & Kennedy 2010). 
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We can take a look at a number of intercultural factors from works of Hofstede (1980), Hampden-Turner 
and Trompenaars (2000) and Faiola and Matei (2005) to provide us with a lens in which to study cultural 
phenomenon. In the interest of brevity, some intercultural variables include individualistic or collectivistic 
cultures, specific or diffuse cultures, low- or high-context cultures, and power management. In addition, the 
use of technology offers an opportunity to leverage what we know about international students to reduce the 
barriers experienced by international students. 

Universal design for learning (UDL) is an attempt to design curriculum with all learners in mind, 
however, suffice it to say that the goal is a lofty one. Nevertheless, the results of keeping a broad perspective 
in mind by applying UDL principles to course design is that if we design for the margins, we are actually 
more inclusive of many more learners. Take for example, a built-in design to provide a virtual office meeting 
for students who wish to understand an assignment with an optional attendance requirement. A virtual 
meeting is ‘live’ which gives students an opportunity to connect with the instructor, ask questions and get 
clarifications. If the session is recorded, other students who could not attend the live session would benefit by 
viewing the recording at a later opportunity. This flexibility supports students with language barriers, 
traditional students who cannot attend at a time specified, and even students who feel uncomfortable seeking 
answers to questions they think they should know. 

2.1 Learner Experiences 

Learner experiences in other countries conflict with the unfamiliar educational culture in the US. 

In many countries, the educational system requires little participation in the classroom and offer limited 
critical thinking opportunities. The teacher is the holder of knowledge and acquisition of knowledge is 
described as memorization. As a result, student challenges include language barriers, weak critical thinking 
skills, differences in cultural communication, and rote learning (Hellsten & Prescott 2004; Yeboah 201 1). 

International students experience both educational and non-educational challenges (Chen and Bennett 
2012; Yeboah 2011). The responsibility is often left to the student to adapt to the host culture (Bevis 2002; 
Lee & Rice 2007) rather than to institutions to accommodate their unique needs. According to Yeboah 
(2011), these challenges result in significant transformations in both education and non-educational settings. 

According to Allen (2010), English language instructors might not know what problems international 
students are facing online. There is a mainstream approach that shoots for designing a curriculum to try to 
accommodate the majority of homogenous learner needs. In fact, international students can experience 
hostility due to their lack of English fluency (Lee & Rice 2007). 

Martirosyan, Hwang, and Wanjohi (2015) indicate that self perceived language proficiency in reading, 
writing, listening, and speaking is a key factor in academic success for international students. In a study by 
Zhang and Kennedy (2010), English language proficiency is the biggest challenge in online courses. Their 
results indicate that the English proficiency requirement needs to be raised to prepare students for online 
classes in the US. This increase in proficiency might be due to the fact that the mostly text-based online 
environment required in an online course requires too much interpretation. 

2.2 Low- Context Communication Style 

Language proficiency plays a significant role in online learning. Consequently, educators should incorporate 
pedagogical strategies that address language barriers in order to help students succeed academically. 

Halls (1976) introduces low and high context communication styles to describe cultural differences. Low 
context is a method of communication requiring explicit and direct language. Low-context communication 
includes facts, words, verbal messages, tasks, directions, and explanations. An LMS where virtual learning is 
occurring is not self-explanatory (Klein & Lalla 2011). While the user interface presents explicit buttons 
labeled in English such as Submit and Attach, there are numerous user interface designs that confiscate such 
as a course starting point and where to go to submit assignments. Other examples of low-context 
communication instances in an LMS include the syllabus, announcements, and the gradebook. 

This author suggests that educators can begin to implement intentional strategies to address language 
barriers by focusing on the learning management system (LMS) and the course syllabus. By first focusing on 
low-context communication, we can concentrate on being intentional about using explicit language to 
improve clarity. 
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2.3 Low- Context Strategies 

The syllabus is usually the first instructional material students access in the online course. As a result, the 
following suggestions about explicit communication are offered to improve clarity in the syllabus: 

• Provide instructor email contact and varied virtual office hours using clearly stated time format 

• Describe course structure and purpose 

• Provide a graphic syllabus to visually represent learning objectives and assignments 

• Describe teaching style and instructor-to-student and student-to-student interaction requirements 

• Give students choice when communicating with instructor (e.g., email, Skype, web conferencing) 

• Include the time zone for all support services, including tech support. Link to everytimezone.com 

• Include academic support services (e.g., tutoring, librarian assistance, and the writing center) 

• Describe and list dates for synchronous requirements if applicable (vary times morning and evening) 

• Indicate instructor response times to communications and to assignments 

• Indicate technology prerequisite skills 

• Describe minimal computer, internet, browser, and technology required for the course 

• Present assignment schedule in advance clearly identifying date/time format you will use 

The LMS offers a range of tools and activities: Left to right menus, words used to describe buttons and 
hyperlinks, branding, and color usage offers mixed messages to the international student. The following 
low-context suggestions are offered to improve learner understanding. 

• Provide the LMS URL prior to beginning of course 

• Hide all LMS tools that will not be used 

• Provide a Getting Started unit to tell students where to begin the course 

• Use features that provide clear and consistent navigation (left to right flow can be confusing) 

• Provide a video with closed captions to explain how to navigate through your course 

• Offer and record an optional ‘live’ virtual meeting to meet students and answer questions 

• Provide video and textual instructions on how to change the interface language of the LMS 

• Provide video and textual instruction on how to use the tools you are using in the LMS 

• Provide video and textual instruction on how to acquire prerequisite technology 

• Provide instructions on how to add a translate button to your browser 

• Use a quiz or writing assessment to evaluate language proficiency 

• Use a survey to inquire about specific learner needs and interests 

• Provide a glossary of terms 

3. CONCLUSION 

Effective teaching practices should be reflected in both designing for and delivering to international students. 
The complexity of intercultural variables makes it seem like a daunting task especially in an online class 
where visual cues offered by face-to-face instruction are unavailable. In addition, the modern learning 
management system is pushing its capabilities to include multiple multimedia tools and flexible user 
interfaces; however, these virtual environments are largely text-based which depend upon a solid proficiency 
in the English language. 

This article focused on language barriers and suggests that educators begin by applying low-context 
strategies in the syllabus and to facilitate LMS use. By focusing on low-context communication issues, U.S. 
educators can take the first step toward designing a culturally inclusive online course that could potentially 
benefit all learners. 
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ABSTRACT 

The diffusion of innovation (DOI) is critical for any organization and especially nowadays for higher education 
institutions (HEIs) in the light of vast pressure of emerging educational technologies as well as of the demand of 
economy and society. DOI takes into account the initial and the implementation phase. The conceptual model of DOI in 
higher education (HE), where e-learning is considered as innovation, will be elaborated in the poster. 


KEYWORDS 
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1. INTRODUCTION 

Europe 2020 Strategy (European Commission, 2010) puts a strong emphasis on smart, sustainable and 
inclusive growth based on innovation and creativity of both enterprises and institutions of higher education 
and science. 

According to Rogers (Rogers, 2003) Diffusion of Innovations (DOI) is the process by which an 
innovation is communicated through certain channels over time among the participants in a social system and 
according to him an innovation is any "idea, practice, or object that is perceived as new by an individual or 
other unit of adoption" (Rogers, 2003). The relevant research question is how to model the DOI related to 
higher education and especially in the context of e-learning implementation. 


2. DIFFUSION OF INNOVATION IN HIGHER EDUCATION 

According to (Rogers, 2003) four main elements influence the spread of an innovation. They are the 
innovation itself, communication channels, time necessary for innovations to be adopted and a social system 
that combines internal and external influences. It also can be recognized that there is the essential influence 
of human capital i.e. there should be a critical mass of people that understand, adopt and spread the 
innovation. Otherwise the sustainability is not ensured. 

The special case of diffusion of innovation in higher education is the implementation of e-learning. Under 
the term of e-learning we comprise all educational technologies starting with technology supported learning 
and blended learning and ending up with the massive open online courses (MOOCs) and personalized online 
learning. According to (Bates, 201 1 ) it is a major innovation in education. 

The innovation process in an organization consists of two main groups of activities (Rogers, 2003): (1) 
Initiation, consisting of information gathering, conceptualization and planning for the adoption of innovation, 
decision to adopt, and (2) implementation, consisting of all the events, actions and decisions involved in 
putting the innovation into use. 

In further research of diffusion of e-learning as an innovation in HE according to the given conceptual 
model (Figure 1) we will define critical factors of DOI for each phase of the innovation process starting with 
the e-readiness assessment of a HE institution and social factors that influence decision making on levels of 
e-learning implementation. 
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After the decision has been made the most influential success indicators for e-learning diffusion into HE 
institution are going to be researched taking into account the following: leadership and strategic alignment, 
dedicated change agents among staff and students, e-learning usefulness recognized by staff and students, 
innovation in teaching, appropriate organization and support structure for staff and students, technical 
infrastructure, quality assurance of e-learning (Begicevic et al, 2007; Bates, 2011; Gon calves & Pedro, 2012). 
The developed model will be upgraded by means of social network analysis (SNA) in order to find key 
communication factors in DOI. 

Finally, the described research will contribute to DOI theory because the majority of the existing research 
is focused on factors that influence an innovation adoption (Vishwanath & Barnett, 201 1 ) and putting aside 
the focal problem of diffusion of innovation through information exchange through communication. 



Figure 1 . Conceptual model of DOI in higher education 

Special issues that influence DOI in higher education are the type of HE institution (private of public); the 
level of autonomy of HE institution, the academic freedom of researchers/professors as well as the 
complexity of fulfilling the usual three-folded mission (education, research and outreach). 


3. CONCLUSION 

The conceptual model of DOI can be tailored for HE institutions but general factors need to be properly 
translated into specific environment, interpreted meaningful and possible connections should be established. 
The recognition of communication agents and channels that enhance the diffusion of e-learning will 
contribute to DOI management in HE system. We are going to present these specifics on the poster. 
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ABSTRACT 

The Digital Resource Exchange About Music (DREAM) is an online tool for exchanging information about digital 
learning tools for music education. DREAM was designed by our team to encourage music teachers to learn about digital 
resources related to learning to play a musical instrument, both in classroom and independent music studio settings. In 
addition to enabling music teachers to learn about resources, DREAM allows music teachers to evaluate the resources 
based on their own experiences using the resources, to read about other teachers’ views of the resources, and to add 
resources to the learning objects repository. During the demonstration of DREAM, we will showcase some of the 
resources and discuss the evolution of the design of DREAM. We will encourage participants to explore DREAM on the 
devices we provide and/or on their own laptops, tablets, or smartphones. 
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1. SESSION DESCRIPTION 

Digital applications for music education have been growing at an astonishing rate, and these resources are 
changing the ways people teach, learn, and make music (Burnard, 2007; Partti, 2012; Rainie & Wellman, 
2012; Waldron, 2013). The Digital Resource Exchange About Music (DREAM) is an online tool especially 
designed for music teachers to help them access the best of the many digital applications that are currently 
available ( www.dreammusictool.ca). DREAM is one of a suite of digital tools for music learning designed by 
a research and development team comprised of members from two Canadian universities, Queen’s University 
and Concordia University, and The Royal Conservatory (RCM), a national conservatory teaching and 
examination system. The first tool in the suite, iSCORE, supports students in their path to becoming self- 
regulated musicians. Two tools that are presently under development include an iOS mobile app for 
annotating music videos called Notemaker, and another web-based tool called Cadenza, which, like iSCORE, 
also was designed to support student self-regulation. 

DREAM allows users to explore digital learning tools in music education, and to add their own tools to 
the digital repository. DREAM provides a forum for teachers to keep informed about the latest websites, 
apps, and recordings that apply to music learning, as well as to exchange their own evaluations of the 
resources. DREAM enables users to download the resources they would like to use directly on their laptops, 
tablets, or smartphones. For example. DREAM can be used to locate games that support sight-reading — a 
perennially difficult area for most music students to master — and to learn about drills to support ear training 
and theory. Users can also listen to excellent recordings of repertoire, and explore professional resources. 
DREAM currently contains over 3,500 high quality resources, all of which have been vetted by professional 
musicians and educators. 

DREAM was designed by our team to encourage music teachers to become familiar with digital resources 
related to learning to play a musical instrument, both in classroom and independent music studio settings. In 
accordance with the other digital tools we have designed, DREAM is an evidence -based tool, created on the 
basis of research that shows how students can become self-regulated learners when their music learning is 
scaffolded appropriately. We have secured funding from the Social Sciences and Humanities Research 
Council of Canada, under the Partnership Grants program, to design and develop tools and to conduct 
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research on the effectiveness of the digital tools in The Suite, and to share those findings with the academic 
community, musicians, teachers, and the general public. Our work is also supported by a Leadership 
Opportunity Grant from the Canadian Foundation for Innovation. While DREAM was designed by Canadian 
developers and researchers, the tool has international reach, especially in English and French speaking 
countries where there is a conservatory system or some other music pedagogy and examination process in 
place. 

In addition to enabling music teachers to learn about resources, DREAM allows music teachers to 
evaluate the resources based on their own experiences using the resources, to read about other teachers’ 
views of the resources, and to add resources to the digital repository. In the release version (DREAM, v. 1.4), 
resources were organized into six categories: (a) musical repertoire, (b) ear training and sight-reading, (c) 
practising, (d) history and theory, (e) creating and composition, and (f) professional resources. All of the 
entries in DREAM are searchable by title and key words, and users can also filter the resources by 
instrument, ability level, or platform (e.g., by type of tablet or smartphone). The resources — whether they are 
recordings, apps, websites, or software programs — are described in the language of origin (English or 
French). 

We characterize DREAM as a Trip Advisor™ for music teachers — but instead of choosing, for example, 
a hotel based on such filters as free parking and internet, the teachers might select musical repertoire, filtering 
for instrument (e.g., violin) and difficulty level (e.g., intermediate). Once the teacher has selected a resource, 
DREAM recommends resources to users based on their prior choices. Again, this has parallels to travel sites, 
for once the user browses through some hotel choices, the user is shown other hotels that might be of interest, 
based on the user’s choice as well as the selections of users with similar profiles. 

The demonstration of DREAM will showcase some of the resources and discuss the evolution of the 
design of DREAM (Upitis, Abrami, Brook, Pickup, & Johnson. 2015). Before releasing DREAM in 
September of 2014, an extensive usability testing period took place. During the first usability study, the 
design and content evolved, based on the feedback from 12 core test participants and designers, a group of 24 
classroom music teachers, as well as nearly 50 studio music teachers from across Canada. The results of this 
usability study provided evidence that DREAM could serve as a centralized place for music teachers to keep 
abreast about digital technologies, by providing a tool that would be valued in terms of efficiency and 
effectiveness. 

The demonstration will also detail some of the patterns of use that have been revealed since DREAM was 
first launched in September of 2014. These will include site analytics on the number of unique users (close to 
5,000), countries of origin, most popular resources, browsers and devices used, and the time spent on the site 
and total page views (over 40,000). 

In addition to describing the design process, patterns of use, and showcasing some of the resources 
contained in DREAM, we will encourage participants to explore DREAM on the devices we provide or on 
their own laptops, tablets, or smartphones. Participants will be able to access resources in the learning objects 
repository, open their own accounts, and add comments to the resources that are already available in 
DREAM. 
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ABSTRACT 

Having classes with “fun” incorporated into their design is crucial for learners. Students can learn from classes that 
combine learning with fun. In this study, we developed a program for university students in a teacher training course that 
aimed to teach ways of incorporating gamification into class design. 
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1. INTRODUCTION 

Having classes with “fun" incorporated into their design is important for learners. Recently, gamification has 
attracted attention as one method of achieving fun in a class design. According to Inoue (2012), 
“Gamification is the act of order to apply the knowledge that has been cultivated in a computer game to the 
reality of social activities.” In this study, we developed a program for university students in a teacher training 
course that aimed to teach ways of incorporating gamification into class design. 


2. DEVELOPMENT AND IMPLEMENTATION OF THE PROGRAM 

In developing the program, we pursued the following three objectives. The first is to develop student teachers’ 
understanding of gamification. The second is to have the student teachers produce an output, namely, create a 
class utilizing gamification. The third is to reduce the lesson time. 

The program, which was developed by us, was conducted in a teacher training class for compulsory 
courses that were prepared by the author. The program was attended by 63 student teachers in their third year 
of studies at the Faculty of Education. 


Table 1. Flow of the program 



Content 

The first class 
(90 minutes) 

Directing and teaching building (D 

• “From dull to fun” “What is gamification?” 

Challenge after class 

Analyze your favorite game from the perspective of gamification. 

The second class 
(90 minutes) 

Directing and teaching building (2) 

• I want to analyze lessons from the viewpoint of production. 

Challenge after class 

To more make the lessons more interesting, design a class conscious lesson in your 
major subject. 
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3. RESULTS 

To clarify the program achievements, we analyzed the participants’ questionnaire responses and free 
descriptions. 

A questionnaire was given both before and after the program. To examine whether the participants had 
acquired the ability to use gamification, we utilized a four-point scale to measure their “interest” and 
“confidence” in lesson planning (4: Very much, 3: Well, 2: A little, and 1: Not much). The analysis included 
the data of 55 student teachers who had participated in both classes, out of the 63 who participated in the 
program. 

A t-test conducted to reveal the average value of the pre- and post-class responses; the responses 
demonstrated a clear and significant difference for the three questionnaire items in terms of “interest” and 
“confidence.” The results are shown in Table 2. 


Table 2. Questionnaire results (n = 55) 


Question 

Before 

After 


(1) I like to devise lesson plans. 

2.24 

2.56 

** 

(2) I have the confidence to devise a fun teaching plan. 

1.89 

2.31 

** 

(3) I have the confidence to make a class interesting. 

2.07 

2.46 

** 


In addition, we analyzed the students’ overall impressions of the lesson in their written responses. 
Consequently, we were able to examine the co-occurrences of the following words: 
“lesson-production-think-guidance” and “study-applications-element.” 

Based on these results, and considering the three objectives of the program, we considered the overall 
effects of the program. First, we discussed the participants’ feelings, experiences, and understanding of 
gamification. In their free descriptions, the participants experienced empathy to notice, as well as the primary 
character of the game and analysis of the game. Utilizing these responses, we effectively analyzed the games 
and stories. 

Second, clearly presenting the aspect that we “produce” class designs with gamification, the students 
demonstrated they were more aware of the connection between gamification and class development. The 
questionnaire results showed that the participants’ “self-confidence to create a fun lesson plan” and 
“self-confidence that the lesson is interesting” improved. Furthermore, in the analysis of the participants’ 
impressions, “lesson-production-think-guidance” showed a strong co-occurrence. Based on these two results, 
the program objectives are considered to have been largely achieved. 

Finally, regarding the teaching time, it became clear from the results that a short time is possible. 

As shown by the above results, the program was effective at developing student teachers’ ability to use 
gamification in class design. 


4. FUTURE DEVELOPMENT 

We have developed a program tool to spread the program. The “lesson plan idea tool” will help teachers 
acquire ideas for incorporating gamification into their class design. In the future, utilizing the idea of a 
program tool, we hope to further develop the program, implement it, and measure its effects in university 
teacher training courses. 
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ABSTRACT 

The purpose of the study is to explore the effects of text -based and non-text-based feedback in a game-based learning 
system. The pilot study is a case study to find low-achieving math students’ comprehension and motivation in using the 
game-based systems. Four low-achieving students from a suburban middle school participated in the study. Data 
collection includes observation and in-depth interview about the use of the GBL. The results will be provided for future 
design and revision the math GBL. 
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1. INTRODUCTION 

Recently, there is a growing concern about increasing number of underachieving students in high schools. 
Students underachieve in mathematics result from many reasons, for example, lack of learning motivation, 
culture deprivation, low SES, or unpleasant learning experiences. A number of remediation instructions and 
strategies have been proposed, basic skill instructions help students to familiar and master skills for reading 
and comprehension, peer tutoring for students to collaborative learning, online learning programs provide 
personal learning resources for students to learn on their own pace inside and outside of classroom. 
Researcher also found that games may be helpful for digital natives. 

Educational researchers suggest that learning is most effective when it is active, experiential, situated, and 
problem-based and provide immediate feedback (Gee, 2007 Prenksky, 2001). Many studies found that 
computer games help learners on knowledge acquisition, improving reasoning, space reasoning, problem 
solving ability as well as motives and attitude (Killi, 2007; Wouters, van Der Spek & Oostendorp, 2009). 
However, some studies found that most of computer games were designed for drill and practice; few of them 
were required to apply special subject content knowledge, or connect to curriculum standards (Ke, 2008). 

Researchers point that feedback is one of fundamental elements of games (Kapp, 2012; Prensky, 2001). 
Studies found that students received immediate, elaborative, text-based feedback led to more effective 
learning and higher motivation (Corbalan, et al., 2010; Kleij, et al., 2012). Another study showed that 
emotional feedback for empathetic encouragement had direct effect on intentions to use CBA (Terzis, et al, 
2012). The proposed game -based learning system is focus on the various types of feedback design. The 
overarching research question is how text-based and non-text based feedback design support low-achieving 
students in mathematics? And what motivate low-achieving students in mathematics engaged in the game- 
based learning system? 


2. METHOD 

A case study method is used to understand the various feedback designs for low-achieving math students. 
Four 7th grade students (2 males and 2 females) are selected from a suburban middle school. Student’s 
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selection is based on their math achievement tests, math teachers’ recommendations and their wiliness to 
participate the study. 

Math Jungle is a game-based learning system for junior high school students to practice algebra 
questions. The GBL includes practice and test mode. In the practice mode, students receive 15 questions. 
Students will read the question stem, and answer the multiple choice types of questions. Students are allowed 
to answer the question twice. When the students answer the questions correctly in the first time, they will 
receive full scores. If they fail in the second time, the correct answers will be presented. During the 
answering questions, students can ask for help. Two types of feedback will be provided. The first type of 
feedback is elaborative, conceptual prompts, the second types of feedback is step-by-step solutions to the 
questions. Students can decide which types of feedback they would like to receive. If they still can’t resolve 
the problem in the second time, the step-by-step solutions will be provided. As the results, points will be 
taken from the total scores. In addition to text feedback, an animated animal (leopard cat) works as an agent 
to guide through the practice mode. The leopard cat will have different facial expression and actions depend 
on the correct or incorrect answers. In the test mode, no prompts or hints will be provided. Students will 
randomly answer 10 questions in the unit without any immediate feedback. All the feedbacks and answers 
will be provided at end of the test mode. A database will record the time and answers that students spend on 
the practice and test mode. The record will provide teachers and parents for further remedial instructions. 

In this pilot test, students will practice the first degree equation as study content. Students will use the 
math jungle GBL in a computer lab individually. Each student will have 50 minutes to finish the practice 
mode. Students can take as much time as they like to practice the 15 questions. Later, students will take the 
test mode to assess their math achievement. After the GBL session, students will take in-depth interview to 
describe their understanding and reaction of the feedback design. 


3. DATA ANALYSIS AND EXPECTED FINDINGS 

To understand the effects of feedback design, students’ answers of math questions will be analyzed. Students 
will be interviewed for their thinking process in the incorrect answers. Student will be required to describe 
their affective reaction toward the GBL as well. The interview transcripts will be analyzed qualitatively. It is 
expected that students are able to demonstrate the problem solving process and use of the feedback in the 
GBL. Game designers and students can discuss the depth and length of the short instruction provided in the 
GBL, and the effects of feedback designs. 
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ABSTRACT 

In conventional information moral education, lessons often focus only on trouble cases and perpetrators without 
examining the level of awareness among the parties involved. This is especially apparent among students who 
communicate over the Internet. For example, one student’s perspective of writing ill on the Internet might greatly differ 
from that of another. Such differences have only confused the Japanese youth regarding the issue. Therefore, this study 
developed an information moral education lesson and conducted a workshop with a group of junior high school students 
to promote awareness and identify any incorrect behaviors that occur while communicating over the Internet. According 
to the survey results, a significant difference was revealed before and after the lesson was conducted. 


KEYWORDS 

Information moral education, Internet communication, Awareness, LINE, Workshops 


1. INTRODUCTION 

According to the “2013 Fiscal Internet Environment Survey of Youth” (Published in 2014), 36.6%, 51.9%, 
and 97.2% of elementary, middle, and high school students possess a mobile phone in Japan, respectively. 
Recently, opportunities for communication via the Internet have increased among Japanese children. In 
particular, the communication tool called “LINE” is currently being used by many children, and such an 
application has made it easier to send messages to friends. However, it has also become a tool to ostracize 
others, which has become the source of trouble among certain Japanese youth (Takeuchi, 2014). As a result 
of such problems, information moral education has been practiced in various regions. For example, some IT 
companies have dispatched employees to schools for presenting lectures and contributing to the development 
of information morals among Japanese children. However, such information moral education tends to focus 
only on trouble cases and perpetrators without examining the level of awareness among the students, 
especially with regard to reviewing their own actions and differentiating between writing negative and 
hateful comments on the Internet. Therefore, this study aims to develop and evaluate an information moral 
lesson that promotes awareness among junior high school students and their communication through the 
Internet. 


2. THEORY AND METHOD 

In the development of teaching and educational materials to promote awareness among students and their 
communication through the Internet, two perspectives have been incorporated. First, it is important to 
recognize the differences between one student’s ideas and that of another’s. Thus, teaching how each student 
has different beliefs and characteristics can be an effective approach to promote awareness among students. 
In the workshop that was developed by the authors, one activity involved comparing various students’ 
opinions and recognizing the differences among them by presenting them in a card format. Second, it is 
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important that students understand the characteristics of communicating over the Internet. For example, it is 
sometimes difficult to understand the other party’s feelings through the use of characters, which can result in 
miscommunication between the sender and the recipient. Therefore, it is important to not only communicate 
but also to consider the other party’s situation. Therefore, we have developed a workshop-type information 
moral lesson for a group of junior high school students (five classes totaling 155 students) in Shizuoka 
Prefecture, Japan. 


3. RESULTS 

The survey provided the students with the following three questions: 1 ) Have you ever harassed a friend? 2) 
Do you think that you wrote ill about friends? and 3) Have you ever contacted your friends late at night? As 
shown in Table 1, a significant difference was observed for each of the items before and after the lesson was 
conducted. 


Table 1. Questionnaire results (average value) 


Question 

Before class 

After class 

Significant difference 

Have you ever harassed a friend? 

1.37 

1.66 

** 

Do you think that you wrote ill about 
friends? 

1.47 

1.60 

* 

Have you ever contacted your friends 
late at night? 

2.38 

2.71 

** 


Based on this result, the information moral lesson that we developed for the students provided a certain 
level of awareness among them. However, it is important to follow up for determining whether this lesson 
had a long-term effect on the participants. 
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ABSTRACT 

In an educational environment, instructors would always think of ways to provide students with motivational learning 
materials and efficient learning strategies. Hence, many researchers have proposed that students’ problem-solving ability 
enhances their learning. Problem-solving ability plays an important role for users in dealing with problems that arise 
during their learning. In order to facilitate high quality performance, computer programmers have been trained to solve 
the problems they encountered. The purpose of this study is to design and develop an e-leaming game-based environment 
integrated with problem-based learning strategies for college freshman majored in computer science. 


KEYWORDS 

Problem based learning. Design-based research. Game-based learning. 


1. INTRODUCTION 

With the advance development of information technology in recent years, more and more electronic devices, 
such as PC, laptop, smart phone or tablet, have been widely used by college students. Many college students 
like to play games on their electronic devices, and digital simulation games have been designed to be more 
visual, interactive and problem-solving focused (Federico & Helene, 2011). Moreover, game-based learning 
(GBL) is a popular and potential learning method in e-learning. Increasingly more research issues have been 
identified. Similar to entertainment and serious gaming, these issues have introduced a new important 
marketing direction to practical technologies such as human-computer interaction, multimedia interaction, 
and ubiquitous computing (Lin, Kinshuk & Mark Dutchuk, 2009). Besides, GBL is attractive to students who 
want to learn the complex/high level knowledge via digital simulation games. 

Therefore, this study is developing a problem-based leaning game based courseware as the learning 
material and use design-based research to evaluate the design and development of this game-courseware, 
which simulated the business management of an e-cafe. This study would like to integrate the principle of 
“usability” to determine whether this courseware corresponds to the above principle and whether it could 
meet the expectations of learners as an education resource (Chang & Huang, 2013). 


2. RELATED WORKS 
2.1 Problem Based Learning 

Problem Based Learning (PBL) focuses on active learning, finding the problems, thinking ability, and the 
process of solving problems. For instructors, it becomes a teaching aid and builds a safe environment for 
students. When PBL is incorporated into a course, it helps student to construct their knowledge concepts and 
integrate their basic knowledge (§endag, & Odaba§i, 2009). In fact, several courses already have PBL 
incorporated into the course design in school settings. One study indicates that PBL incorporated into the 
course design helps students focus more on the connection between real problems and subject knowledge. 
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Furthermore, using PBL in online courses also help students improve their critical thinking (Cheng, She, & 
Annetta, 2014). 

2.2 Design based Research 

Design-based research (DBR) is a research methodology and also known as design experiments. It is defined 
as “an emerging paradigm for the study of learning in content through the systematic design and study of 
instructional strategy and tools” (Design-Based-Research Collective, 2003, p. 5). DBR approach is an 
iterative cycle of multiple steps, including design exploration, interventions enactment, outcome evaluation 
and analysis, and redesign (Chang, & Wong, 2014; Wong, Boticki, Sun, & Looi, 2011). Based on the 
research (Hartson, Andre & Williges, 2003), usability evaluations in this study is tied to each phase of the 
DBR, the following figure 1 is the original design of pbl game-based courseware development process 
adopted from DBR. 
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identify training needs 
(interview and questionnaire) 
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Product release 
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(summative evaluation! 

S. J 
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V / 
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Revision 

\ ; 
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(usability evaluation) 


I 

I 

I 

< Implementation 

Beta: effectiveness evaluation 
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Figure 1. PBL game-based courseware development process adopted from DBR 


3. CONCLUSION & FUTURE WORK 

In this research, we propose a pbl game-based courseware called Programmers’ Adventure, which were 
tested by 65 college freshman majored in computer science in an university, Taiwan. The Programmers’ 
adventure courseware is designed for users to familiarize with knowledge of programming language. Based 
on the results of pilot study, students’ subject knowledge about program language in application - cognitive 
process dimension have improved. In addition, game designers received positive feedback from students 
regarding the Programmers Adventure game. Finally, our research will test the usability of the Programmers 
Adventure game -based courseware. The results will give us several improvement suggestions in the future, 
and we hope to provide a well -designed game-based courseware for students to help them learn efficiently 
more complex/high level knowledge. 
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